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ABSTRACT 


E-learning  has  been  defined  as  “the  use  of  network  technologies  to  create,  foster, 
deliver,  and  facilitate  learning,  anytime  and  anywhere”  (Einezine,  2000).  E-learning  is 
one  of  the  fastest  growing  services  on  the  Internet  today  with  revenues  between  6  billion 
and  7  billion  dollars  (Bizreport,  2003).  This  growth  comes  with  a  peculiar  problem  - 
many  students  never  complete  their  e-learning  courses.  Although  there  is  significant 
variation  among  institutions  —  with  some  reporting  course-completion  rates  of  more  than 
80  percent  and  others  finding  that  fewer  than  50  percent  of  distance-education  students 
finish  their  courses  ...  course-completion  rates  are  often  10  to  20  percentage  points 
higher  in  traditional  courses  than  in  distance  offerings.”  (Carr,  2000). 

There  have  been  few  credible  studies  investigating  actual  non-completion  rates 
and  factors  that  affect  these  dropout  rates  (O’Connor  et.al.,  2003).  This  study  seeks  to 
apply  Human-Computer  Interaction  theory,  through  use  of  a  usability  inspection  method 
called  Heuristic  Evaluation,  to  identify  course  design  characteristics  that  may  influence 
course  completion  rates.  A  research  instrument  was  developed  and  applied  to  twenty  (20) 
courses  offered  by  the  Air  Eorce  Institute  of  Technology’s  School  of  Systems  and 
Eogistics,  and  a  Pearson  Correlation  was  performed  to  identify  any  relationships  between 
design  factors  and  course  completion  rates.  Analysis  indicated  some  support  for  using  the 
Heuristic  Evaluation  method.  Practical  and  theoretical  implications  for  this  research  are 
also  discussed. 
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A  STUDY  OF  COURSE  DESIGN  EACTORS  THAT  INFEUENCE  E-EEARNING 


COURSE  COMPLETION  RATES 

I.  Introduction 

“What  I  admire  in  Columbus  is  not  his  having  discovered  a  world  but  his  having  gone  to 

search  for  it  on  the  faith  of  an  opinion.  ” 

A.  Robert  Turgot 


Background 

E-learning  has  been  defined  as  “the  use  of  network  teehnologies  to  ereate,  foster, 
deliver,  and  faeilitate  learning,  anytime  and  anywhere”  (Linezine,  2000).  E-learning  is 
one  of  the  fastest  growing  serviees  on  the  Internet  today  with  revenues  between  6  billion 
and  7  billion  dollars  (Bizreport,  2003).  This  pervasive  nature  of  the  Internet  has  ereated  a 
golden  opportunity  for  corporations  and  educational  institutions  to  provide  knowledge 
and  learning  around  the  globe  using  e-learning  services.  The  federal  government  has  also 
taken  a  keen  interest  in  e-learning.  White  House  Executive  Order  #13111  (1999)  requires 
all  executive  agencies  to  identify  one  area  of  training  and  implement  an  e-learning 
solution.  The  intent  of  this  order  is  to  increase  awareness  and  use  of  technology  to 
provide  government  employees  with  more  training  opportunities. 

In  the  midst  of  this  enthusiasm,  there  is  a  continuing  concern  over  e-learning 
course  completion  rates. 

Although  there  is  significant  variation  among  institutions  —  with  some  reporting 

course-completion  rates  of  more  than  80  percent  and  others  finding  that  fewer 
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than  50  percent  of  distance-education  students  finish  their  courses  . . .  course- 
completion  rates  are  often  10  to  20  percentage  points  higher  in  traditional  courses 
than  in  distance  offerings.”  (Carr,  2000). 

A  study  by  Reynolds  (2002)  looked  at  this  problem  for  the  Air  Force  Institute  of 
Technology  School  of  Systems  and  Logistics  Virtual  Schoolhouse  (VSH)  where 
completion  rates  were  very  close  to  the  range  described  by  Carr  with  completion  rates 
ranging  between  48  percent  and  79  percent.  Reynolds  investigated  the  VSH’s  low 
completion  rate  problem  from  a  motivational  perspective,  identifying  such  factors  as 
completion  goals,  off-task  distractions,  availability  of  feedback  for  self-regulation,  and 
continued  confidence  were  important  factors  that  distinguished  those  who  completed 
their  courses  from  those  who  did  not.  Reynolds  also  identified  problems  with  course 
design  like  lack  of  readability  and  excessive  course  length  as  possible  contributing 
factors.  His  results  were  not  statistically  conclusive  -  mainly  because  of  a  small  sample 
size,  the  present  study  attempts  to  correct  this  problem  and  then  expand  on  the  potential 
course  design  characteristics  may  influence  VSH  course  completion  rates. 

There  has  been  much  research  on  the  motivational  factors  influencing  e-leaming 
completion  rates,  but  less  research  exploring  the  effects  of  course  design  on  course 
completion  rates.  A  study  by  researchers  at  James  Madison  University  that  surveyed  a 
group  of  e-learning  managers  from  various  organizations  and  industries  found  that 
instructional  design-related  factors  and  lack  of  motivation  had  an  equal  effect  on  course 
completion  (O’Connor  et.ak,  2003).  This  would  indicate  that  problems  with  course 
design  have  as  much  effect  on  completion  rates  as  learner  motivation. 
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One  key  eonsideration  in  designing  an  e-learning  course  is  to  separate  the  needs 
of  a  user  from  the  needs  of  a  learner.  (Smulders,  2004)  differentiates  these  types  based  on 
the  operator  consciousness  of  the  web-based  technology.  Users  interact  with  the  form  of 
the  Web  course,  the  interface,  navigation,  information  architecture,  and  the  visual  design 
of  both  the  screen  layout  and  the  actual  content.  A  learner,  on  the  other  hand,  is  a  user 
who  can  easily  navigate  their  way  through  the  form  of  the  Web-based  environment  and 
access  the  content.  The  design  of  the  course  becomes  transparent  and  the  learner  begins 
interacting  with  the  content,  instructors,  and  fellow  students,  and  finally  begins  to  learn. 
Identifying  and  addressing  design  factors  that  hinder  the  student’s  use  of  the  e-leaming 
system  should  help  make  the  transition  from  user  to  learner  easier  and  faster. 

Human-computer  interaction  (HCI)  theory  provides  readily  available  constructs  in 
the  form  of  usability  inspection  methods  that  should  allow  e-learning  course  designers  to 
help  users  become  learners.  One  usability  inspection  method  that  is  particularly  appealing 
is  heuristic  evaluation  which  involves  having  a  small  set  of  evaluators  examine  the 
interface  and  judge  its  compliance  with  recognized  usability  principles  (the  “heuristics”) 
(Nielsen,  1994).  The  method’s  appeal  stems  from  its  ability  to  identify  many  usability 
problems  while  being  easy  to  learn  and  relatively  inexpensive  to  use.  Using  heuristic 
evaluation  to  identify  usability  factors  that  inhibit  students  from  easily  accessing  course 
materials,  and  thereby  interfering  in  the  learning  process,  may  provide  insights  into  why 
some  individuals  complete  e-learning  courses  while  others  drop  out. 

Unfortunately,  given  the  anecdotally-claimed  wide  range  of  non-completion  rates 
for  e-leaming  in  the  popular  press  and  trade  journals,  there  have  been  few  credible 
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studies  investigating  actual  non-completion  rates  and  factors  that  affect  these  dropout 
rates  (O’Connor  et.al.,  2003).  According  to  e-Leam  magazine,  “the  idea  of  e-leaming 
usability  is  still  so  new  it  is  barely  on  the  market’s  radar  screen.  When  the  market 
eventually  insists  on  usability  testing,  e-leaming  usability  experts  will  be  the  ones  making 
sure  e-learning  courseware  is  as  effective  as  it  is  stylish— and  the  data  they  uncover  may 
offer  insights  on  the  high  dropout  rates  from  online  education  courses”  (2004:2). 

Because  e-leaming  and,  therefore,  the  study  of  e-learning  usability  are  such  recent 
developments,  more  studies  are  needed  to  create  a  body  of  knowledge  that  can  direct 
further  research. 

Research  Focus 

The  Air  Force  Institute  of  Technology  School  of  Systems  and  Logistics  is 
concerned  over  the  apparent  low  completion  rates  for  some  courses  offered  through  their 
Virtual  Schoolhouse.  This  study  will  look  to  HCI  usability  factors  to  explain  variation  in 
course  completion  rates,  as  well  as  build  upon  the  work  of  Reynolds  by  retesting 
hypotheses  related  to  course  length  and  readability  of  course  material  using  an  expanded 
data  analysis  that  covers  20  VSH  courses.  Like  most  software  development,  creation  and 
maintenance  of  e-learning  courses  can  be  very  expensive.  Knowing  where  to  focus 
design  efforts  can  help  e-learning  course  designers  and  developers  reduce  these  costs 
while  providing  a  better  product  in  return. 
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Implications 

This  study  will  have  both  practical  and  theoretical  implications  for  researchers 
looking  at  the  low  e-leaming  course  completion  rate.  Practical  implications  include 
identification  of  maintenance  problems  such  as  identifying  broken  or  misdirected 
hyperlinks,  interactive  content  that  is  difficult  to  use,  and  spelling  and  grammatical 
errors.  Theoretical  implications  for  this  study  include  development  of  a  heuristic 
evaluation  checklist,  tailored  to  e-learning  courses,  that  is  derived  from  work  in  usability 
testing  of  graphical  user  interfaces. 

Thesis  Overview 

This  chapter  has  provided  a  brief  introduction  to  the  issues  confronting  e-learners 
that  course  designers  must  consider  when  developing  e-leaming  courses.  Chapter  II 
expands  on  work  by  Reynolds  (2002),  and  builds  a  case  for  evaluating  course  design 
through  heuristic  methods  as  an  approach  to  explain  low  e-learning  course  completion 
rates.  Chapter  III  presents  the  methodology  used  to  test  the  hypotheses,  describes  the 
development  of  the  instmment  used  to  evaluate  the  VSH  courses,  and  explains  the  data 
collection  techniques.  Chapter  IV  presents  the  statistical  analysis  of  the  data  collected. 
Chapter  V  presents  the  conclusions  reached  from  this  study  and  recommendations  for 
further  research  in  this  area. 
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II,  Literature  Review 


“When  you  take  stuff from  one  writer,  it ’s  plagiarism;  but  when  you  take  it  from 
many  writers  it ’s  research.  ” 

Wilson  Mizner 

Introduction 

This  chapter  begins  with  an  overview  of  the  Air  Foree  Institute  of  Teehnology, 
School  of  Systems  and  Logistics  Virtual  Schoolhouse.  The  overview  includes  the  goal, 
structure,  and  operation  of  the  VSH.  Next,  literature  from  usability  inspeetion  methods 
of  user-interfaee  design  is  reviewed  to  identify  system  design  eharacteristies  that  may 
diseourage,  or  prevent,  students  from  eompleting  an  e-learning  course.  Specifie 
hypotheses  are  stated  to  deseribe  the  relationship  between  the  heuristic  and  course 
completion  rates. 

Virtual  Schoolhouse 

The  Air  Foree  Institute  of  Teehnology  (AFIT)  Sehool  of  Systems  and  Logistics 
Virtual  Schoolhouse  is  a  web-based,  asynchronous  learning  environment  with  the 
mission  of  providing  eontinuing  education  resources  to  the  student’s  desk,  on  the 
student’s  sehedule  (The  Department  of  Systems  Aequisitions  AFIT  Virtual  Sehoolhouse 
homepage,  2004).  The  VSH  offers  over  20  web-based  courses  in  sueh  areas  as 
aequisition,  configuration  management,  software  engineering,  sustainment,  and  systems 
engineering  to  United  States  Air  Foree  and  other  Department  of  Defense  personnel 
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working  in  the  acquisition  management  and  logistics  career  fields.  While  VSH  is 
designed  to  allow  students  freedom  in  when  and  where  they  complete  a  course,  certain 
time  restrictions  are  placed  on  the  courses  to  ensure  the  availability  of  an  instructor 
should  the  student  have  any  questions  on  the  use  of  the  VSH  system  or  on  the  material 
presented  in  the  course. 

VSH  courses  are  not  mandatory  for  certification  of  acquisition  personnel,  nor  do  they 
provide  any  academic  credit.  The  VSH  courses  are,  however,  an  important  source  of 
continuous  learning  for  acquisition  personnel.  A  memorandum  issued  by  the  Under 
Secretary  of  Defense  (Acquisition,  Technology,  and  Logistics)  requires  all  acquisition 
personnel,  whether  certified  or  not,  to  obtain  80  continuous  learning  points  every  two 
years  (Under  Secretary  of  Defense  (AT&L),  2003).  VSH  courses  offer  credits  that  can  be 
applied  toward  fulfilling  this  requirement.  According  to  a  VSH  administrator,  the  school 
is  finding  it  difficult  and  expensive  to  maintain  courses.  The  VSH  administrator  believes, 
completion  rates  have  dropped  in  recent  years  from  nearly  80%  to  67%  in  calendar  year 
2002  as  a  result  of  these  maintenance  issues.  In  order  to  address  the  low  course 
completion  rates,  VSH  developers  and  administrators  are  looking  for  tools  that  will  allow 
them  to  continually  assess  the  courses  and  design  effectiveness.  One  family  of  tools  that 
might  prove  helpful  in  these  assessments  is  usability  inspection. 

Usability  Inspection  and  Heuristic  Evaluation 

Usability  Inspection  (UI)  is  a  sub-discipline  of  the  Human-Computer  Interaction 
field,  which  has  roots  in  numerous  disciplines  including  computer  graphics,  operating 
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systems,  human  factors,  ergonomics,  industrial  engineering,  cognitive  psychology,  and 
the  systems  part  of  computer  science  (Hewett,  Baecker,  Card,  Carey,  Gasen,  Mantei, 
Perlman,  Strong,  and  Verplank,  2003).  UI  is  the  generic  name  of  a  set  of  methods  based 
on  having  evaluators  inspect  or  examine  usability-related  aspects  of  a  user  interface 
(Mack  and  Nielsen,  1994).  One  usability  inspection  method  that  is  cost  effective  and  easy 
to  use  is  Heuristic  Evaluation  (Nielsen,  1994).  The  Heuristic  Evaluation  method  involves 
a  small  group  of  evaluators  who  inspect  an  interface  and  judge  its  compliance  with  some 
set  of  recognized  usability  principles  -  the”heuristics”  (Nielsen,  1994:28).  Nielsen 
developed  an  original  list  of  general  heuristics  that  evaluators  can  choose  from  to  meet 
the  requirements  of  the  evaluation  being  conducted.  In  addition  to  Nielsen’s  heuristics, 
practitioners  at  the  Online  Computer  Eibrary  Center  (OCEC)  have  defined  additional 
heuristics  to  evaluate  their  online  systems  (OCEC,  2003). 

A  heuristic  describes  qualities  or  characteristics  that  are  a  part  of  a  usable 
interface.  To  help  guide  evaluators  during  an  inspection,  Piero tti  (2002)  developed  a 
checklist  of  items,  or  characteristics,  for  each  of  Nielsen’s  heuristics,  that  evaluators 
should  look  for  to  determine  how  well  a  system  conforms  to  the  heuristic.  Many  of  these 
items  can  be  applied  to  the  design  of  VSH  courses.  Nine  heuristics  and  the  hypothesized 
relationships  for  VSH  courses  are  summarized  in  the  paragraphs  below.  The  first  seven 
stem  primarily  from  Nielsen’s  revised  set  of  usability  heuristics  and  use  characteristics 
tailored  from  Pierotti’s  checklist.  The  last  two  are  taken  from  the  OCEC  heuristics. 

These  nine  heuristics  are  depicted  in  Eigure  1  on  the  next  page. 
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Visibility  of  System  Status 

Nielsen  (1994)  asserts  that  the  system  should  always  keep  the  user  informed  about 
what  is  going  on,  through  appropriate  feedback  within  reasonable  time.  Every  display 
should  present  the  user  with  a  title  or  header  that  describes  the  screen  contents  (Pierotti, 
2002).  If  pop-up  windows  are  used  to  display  error  messages,  the  window  should  not 
obscure  the  field  the  error  message  pertains  to  (Pierotti,  2002).  Other  important 
characteristics  include  some  sort  of  system  feedback  for  every  operator  action,  which 


Figure  1.  Relationship  of  Heuristic  Constructs  to  Course  Completion. 
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can  be  in  the  form  of  an  indieation  that  the  next  module  ean  be  started  when  the  eurrent 


module  has  been  eompleted,  a  visual  indieation  of  where  to  plaee  objects  that  are  to  be 
moved,  or  a  message  to  the  user  informing  them  of  what  is  going  on  when  there  are 
lengthy  delays  in  system  proeessing  (Piero tti,  2002).  These  feedbaek  eharaeteristics  are 
present  in  VSH  eourses  to  varying  degrees  exeept  for  displaying  a  message  during 
lengthy  delays.  The  lack  of  these  messages  may  eause  some  users  to  believe  the  course 
has  stopped  responding,  leading  them  to  leave  the  eourse  prematurely. 

Other  system  status  indieators  inelude  eontext  labels,  menu  maps,  and  plaee 
markers  that  aet  as  navigational  aids  when  the  user  must  navigate  between  multiple 
sereens  (Pierotti,  2002).  These  plaee  markers  traek  a  student’s  progress  through  a 
module,  indicating  how  much  they  have  aeeomplished  and  how  mueh  remains.  Plaee 
markers  ean  also  help  users  make  a  deeision  on  whether  to  stop  in  the  middle  of  a  module 
or  to  finish  before  exiting  the  VSH  system.  The  effeet  of  these  system  status 
eharaeteristies  on  course  eompletion  leads  to  the  first  hypothesis. 

Hypothesis  1:  There  will  be  a  positive  relationship  between  the  visibility  of 

system  status  within  a  eourse  and  eourse  eompletion  rates. 

Match  between  the  System  and  the  Real  World 

The  system  should  speak  the  user’s  language,  with  words,  phrases,  and  coneepts 
familiar  to  the  user,  rather  than  system-oriented  terms  (Nielsen,  1994).  Courses  that  use 
many  unfamiliar  terms  and  eomplex  sentenees  will  be  more  diffieult  to  use  than  eourses 
that  use  familiar  words  in  simple  sentenees.  Reynolds  (2002)  tapped  into  this  heuristie  by 
measuring  eourse  difficulty  through  readability  using  a  simple  tool,  ealled  the  Flesch 
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Reading  Ease  (Flesch,  1962).  This  tool  is  readily  available  through  Mierosoft  Word  and 
is  widely  used  by  currieulum  developers  to  match  the  difficulty  of  course  materials  with 
the  reading  ability  of  the  students  who  will  be  using  it.  Another  tool  readily  available 
through  Microsoft  Word  is  the  Flesch-Kincaid  Grade  level.  This  tool  converts  the 
readability  score  to  a  United  States  grade-school  level  (Anderson,  2003).  For  example,  a 
score  of  8.0  means  that  an  eighth  grader  can  understand  the  document.  These  quantitative 
measures  may  be  helpful  in  identifying  an  aspect  of  the  VSH  courses  that  is  contributing 
to  lower  course  completion  rates  and  lead  to  the  second  hypothesis. 

Hypothesis  2:  There  will  be  an  inverse  relationship  between  the  difficulty  of 

course  materials  and  course  completion  rates. 

User  Control  and  Freedom 

According  to  Nielsen  (1994),  users  should  be  free  to  select  and  sequence  tasks 
(when  appropriate),  rather  than  have  the  system  do  this  for  them.  In  courses  that  use 
overlapping  windows,  it  should  be  easy  for  the  user  to  rearrange  the  windows  on  the 
screen,  be  able  to  switch  between  the  windows,  and  each  window  should  allow  both 
vertical  and  horizontal  scrolling  (Piero tti,  2002).  As  new  windows  appear  on  the  screen, 
users  can  become  disoriented  and  lose  track  of  where  they  were  before  the  new  windows 
appeared.  This  disorientation  can  cause  the  user  to  become  frustrated  while  trying  to  find 
where  he/she  came  from.  If  there  are  multiple  menu  levels,  the  system  should  provide  a 
mechanism  to  allow  the  user  to  go  back  to  previous  menus,  and  if  a  question  and  answer 
interface  is  used,  the  user  should  be  able  to  go  back  to  previous  questions  or  skip  ahead  to 


11 


later  questions  (Pierotti,  2002).  The  effect  of  these  user  control  and  freedom 
characteristics  on  course  completion  leads  to  the  third  hypothesis. 

Hypothesis  3:  There  will  be  a  positive  relationship  between  the  level  of  user 

control  and  freedom  within  a  course  and  course  completion  rates. 

Consistency  and  Standards 

Nielsen  also  suggested  that  users  should  not  have  to  wonder  whether  different 
words,  situations,  or  actions  mean  the  same  thing  (1994).  Formatting  standards  should  be 
followed  consistently  in  all  screens  within  a  course  (Pierotti,  2002).  Because  VSH 
courses  are  presented  over  the  World  Wide  Web,  users  may  expect  Web  conventions  to 
apply  to  the  course  material.  An  example  of  this  would  be  underlined  text.  On  the  World 
Wide  Web,  underlined  text  is  used  to  identify  a  hyperlink,  so  a  user  will  expect  a  new 
Web  page  to  appear  when  they  click  on  the  text  and  could  become  confused  if  nothing 
happens. 

Other  characteristics  to  look  for  when  evaluating  consistency  and  standards 
include  labeling  of  icons,  especially  system  specific  icons  as  they  may  be  unfamiliar  to 
the  user,  and  careful  use  of  attention-getting  techniques  (Pierotti,  2002).  When  the  system 
asks  for  a  specific  answer  to  a  question  and  more  than  a  one  word  answer  is  required,  the 
valid  inputs  should  be  listed  so  the  user  will  know  how  the  system  expects  them  to  enter 
their  response  (Pierotti,  2002).  The  effect  of  these  consistency  and  standards 
characteristics  on  course  completion  leads  to  the  fourth  hypothesis. 

Hypothesis  4:  There  will  be  a  positive  relationship  between  the  platform 

consistency  and  standards  within  a  course  and  course  completion  rates. 
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Aesthetic  and  Minimalist  Design 

Nielsen  (1994)  also  suggests  that  dialogues  should  not  contain  information  which 
is  irrelevant  or  rarely  needed.  Every  extra  unit  of  information  in  a  dialogue  competes  with 
the  relevant  units  of  information  and  diminishes  their  relative  visibility  (Nielsen,  1994). 

A  screen  should  only  display  information  essential  to  understanding  the  current  topic  or 
to  any  decision  that  the  user  must  make  (Piero tti,  2002).  While  courses  that  are 
entertaining  can  be  more  interesting,  VSH  course  designers  should  be  sure  the 
information  the  user  needs  is  not  being  lost  in  the  presentation.  Color  should  be  used  with 
discretion  and  menu  titles  should  be  brief,  yet  long  enough  to  communicate  their  meaning 
to  the  user  (Piero tti,  2002).  The  effects  of  these  aesthetic  and  minimalist  design 
characteristics  on  course  completion  leads  to  the  fifth  hypothesis. 

Hypothesis  5:  There  will  be  a  positive  relationship  between  aesthetic  and 

minimalist  design  characteristics  within  a  course  and  course  completion  rates. 

Help  and  Documentation 

Even  though  it  is  better  if  the  system  can  be  used  without  documentation,  Nielsen 
suggests  that  it  may  be  necessary  to  provide  help  and  documentation,  and  any  such 
information  should  be  easy  to  search,  focused  on  the  user’s  task,  list  concrete  steps  to  be 
carried  out,  and  not  be  too  large  (1994).  The  help  function  should  be  visible  on  every 
screen  within  a  course,  it  should  be  easy  to  access  and  return  from  the  help  system,  and 
the  information  provided  by  the  help  system  should  be  relevant  to  the  user’s  task 
(Piero tti,  2002).  Help  files,  such  as  a  list  of  acronyms  used  within  a  course,  that  a  user 
can  access  from  every  page  may  reduce  the  time  a  user  spends  searching  for  the  first  use 
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of  an  acronym.  Acronyms  that  are  unfamiliar  to  the  user  may  need  to  be  refereneed 
numerous  times  before  they  are  easily  reeognized.  The  effeets  of  theses  help  and 
doeumentation  eharaeteristics  on  course  eompletion  leads  to  the  sixth  hypothesis. 

Hypothesis  6:  There  will  be  a  positive  relationship  between  the  level  of  help  and 

doeumentation  within  a  course  and  course  completion  rates. 

Skills 

Beyond  the  other  eharaeteristics,  Nielsen  states  in  his  Flexibility  and  Efficiency  of 
Use  heuristie  that  the  system  should  support,  extend,  supplement,  or  enhance  user’s 
computer  skills,  background  knowledge,  and  expertise  -  not  replace  them  (1994).  Pierotti 
(2002)  renames  this  heuristie  to  “Skills”  and  this  name  will  be  used  throughout  this  study. 
Pierotti  (2002)  also  states  that  window  operations  should  be  easy  to  learn  and  use,  and  the 
system  should  provide  support  for  noviee  and  expert  users,  and  that  the  eursor  should  be 
positioned  in  the  textbox  when  the  user  enters  a  sereen  that  contains  one.  Users  should 
not  have  to  spend  more  time  learning  how  to  use  the  course  software  than  they  spend 
learning  the  eourse  material,  nor  should  they  have  to  make  unnecessary  control  inputs. 

The  effeets  of  a  user’s  computer  skills  on  course  completion  leads  to  the  seventh 
hypothesis. 

Hypothesis  7:  There  will  be  a  positive  relationship  between  the  level  of  support 

for  the  user’s  skills  within  a  course  and  eourse  completion  rates. 

Use  of  Chunking 

To  Nielsen’s  seven  heuristics,  the  Online  Computer  Library  Center  adds  that 
material  should  be  written  so  that  doeuments  are  short  and  contain  exactly  one  topic 
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(OCLC,  2003).  The  concept  of  chunking  is  not  unlike  Taylor’s  (1917)  concepts  of  task 
specialization,  and  time  study.  In  fact,  Taylor  uses  the  example  of  a  teacher  breaking  a 
long  lesson  into  daily  tasks  so  the  students  can  accomplish  the  work  in  a  specific  amount 
of  time.  The  trick  in  applying  Taylor’s  concepts,  using  the  example  of  the  teacher’s  long 
lesson,  is  in  determining  how  much  of  the  task  a  student  should  work  before  requiring  a 
break. 

In  applying  these  concepts  to  the  Virtual  Schoolhouse  courses,  a  task  would  be 
defined  as  a  module,  and  the  job  of  course  designers  would  be  to  size  modules  in  such  a 
way  that  students  are  able  to  complete  them  before  needing  a  break  or  being  pulled  away 
from  completing  the  module  by  other  demands.  This  study  makes  no  attempt  at  trying  to 
determine  what  the  optimal  length  of  a  module  should  be.  Rather,  this  study  only 
attempts  to  determine  if  the  current  average  module  length  has  a  negative  linear 
relationship  to  course  completion  rates. 

Hypothesis  8:  The  average  length  of  the  modules  within  VSH  courses  will  have 

an  inverse  relationship  to  course  completion  rates. 

Don ’t  Lie  to  the  User 

The  final,  and  possibly  most  important  heuristic  is  the  elimination  of  erroneous  or 
misleading  hyperlinks.  The  OCLC  warns  designers  to  not  refer  to  missing  information 
(2003).  A  hyperlink  on  a  World  Wide  Web  page  is  an  embedded  command  that  appears 
as  text  within  the  document.  E-learning  course  developers  often  use  hyperlinks  to  provide 
access  to  background  information  or  supporting  documentation  for  some  topic  presented 
in  the  course.  When  activated  (clicked  on),  the  hyperlink  command  instructs  the  browser 
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to  retrieve  and  display  a  different  Web  page.  By  activating  the  hyperlink,  an  e-leamer  has 
acted  upon  a  need  for  the  information  represented  by  that  link.  Repeated  failure  of  the  e- 
learner  to  access  information  may  begin  to  erode  his  or  her  confidence  in  the  usefulness 
of  the  course  as  a  reliable  source  of  information.  Reynolds  (2002)  looked  at  software 
problems  such  as  broken  hyperlinks  and  found  they  did  not  appear  to  have  a  significant 
impact  on  an  e-learners  motivation  to  complete  a  course.  However,  Reynolds’  analysis 
was  based  on  comments  from  course  critiques,  which  are  normally  filled  out  by  students 
after  they  complete  a  course.  The  use  of  course  critiques  raises  the  question  of  non- 
contact  bias  since  it  seems  probable  that  a  student  who  fails  to  complete  a  course  will  also 
fail  to  submit  a  course  critique.  By  looking  at  the  correlation  between  broken  hyperlinks 
and  actual  course  completion  rates,  this  study  should  overcome  this  bias.  The  effect  of 
broken  hyperlinks  on  course  completion  leads  to  the  ninth  hypothesis. 

Hypothesis  9:  The  percentage  of  broken  hyperlinks  within  a  course  will  have  an 

inverse  relationship  to  course  completion  rates. 

Summary 

This  study  used  existing  literature  to  identify  heuristics  that  may  influence  course 
completion  rates.  Nine  distinct  guidelines  have  been  offered  as  possible  factors  that  may 
influence  course  completion  rates  with  the  objective  to  create  a  checklist  tailored  to 
develop  and  maintain  web-based,  asynchronous  courses.  The  following  chapter  describes 
the  methodology  followed  to  test  the  nine  hypotheses. 
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III.  Methodology 


”A  leading  authority  is  anyone  who  has  guessed  right  more  than  once.  ” 

Frank  A.  Clark 

Introduction 

This  chapter  describes  the  methodology  used  to  measure  design  factors  that 
influence  e-learning  eourse  eompletion  rates.  A  literature  review  was  used  to  develop  the 
research  instrument  and  to  identify  other  measures  for  this  study.  The  research 
instrument,  entitled  the  Heuristie  Evaluation  Cheeklist  (HEC)  (see  Appendix  A),  was 
modeled  after  a  similar  eheeklist  developed  by  Pierotti  (2002)  and  deseribed  in  Chapter 
II.  The  other  measures  ehosen  were  the  Eleseh  Reading  Ease  (Eleseh,  1962),  the  Eleseh- 
Kincaid  Grade  Eevel  (Anderson,  2003),  and  a  word  eount  to  measure  eourse  length. 
Twenty  (20)  eourses  from  the  Air  Eoree  Institute  of  Teehnology  Virtual  Sehoolhouse 
were  chosen  for  analysis.  The  following  sections  describe  the  course  charaeteristics, 
instrument  development,  data  eolleetion  proeedures,  and  statistieal  analysis  used  in  this 
research  effort. 

Sample  Courses 

Twenty  eourses  from  the  Virtual  Sehoolhouse  were  selected  for  this  study.  These 
eourses  were  all  of  the  eourses  available  at  the  time  of  this  study.  They  eover  a  variety  of 
aequisition  related  topics  and  have  a  fairly  wide  range  of  eourse  eompletion  rates  (49 
pereent  to  78  pereent).  There  were  a  total  of  12,234  enrollments  for  these  eourses  from 
CY  2000  to  CY  2003.  The  average  eompletion  rate  aeross  all  twenty  eourses  was  about 
67  pereent.  Table  1  shows  the  distinguishing  eharaeteristies  for  the  sample  eourses. 
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Table  1.  Characteristics  of  Target  Courses 


Course  Name 

Course 

Number 

Course  Description 

Number 

Enrolled 

Number 

Completed 

E„ 

Activity  Based 

Costing 

FIN  160 

Introduces  ABC  and  discusses  AFMC  strategic 
planning  process. 

977 

479 

49.0 

Introduction  to  Risk 
Management 

SYS  165 

Presents  elements  of  risk  management  in  acquisition. 

1293 

673 

52.0 

Advanced  Concepts 

Technology 

Demonstration 

SYS  190 

Provides  information  to  facilitate  ACTD  project 
support. 

435 

259 

59.5 

Earned  Value 
Management  System 

FIN250 

Instructs  on  applying  EVM  to  evaluate  contractor 
performance. 

1093 

654 

59.8 

Reformed  Supply 
Support  Program 

SYS  180 

Educates  students  in  a  reengineering  process  for  supply 
support. 

489 

305 

62.4 

Intelligence  in  Force 
Modernization 

SYS031 

Addresses  the  role  of  the  Intelligence  community  in 
acquisition. 

329 

206 

63.4 

Current  Topics  in 
Financial 

Management 

FIN150 

Presents  students  with  current  Financial  Management 
facts  and  principles. 

1118 

718 

64.2 

Incentives  for 

Reducing  Acquisition 
Response  Time 

SYS352 

Addresses  incentives  to  reduce  acquisition  cycle  time 
on  weapon  system  programs. 

141 

95 

67.4 

Introduction  to 

Configuration 

Management 

SYS028 

Presents  configuration  management  facts  and  principles 
in  the  light  of  current  acquisition  reform  practices. 

1002 

688 

68.7 

Maintenance  Planning 
Course 

SYS  170 

Covers  acquisition  policy  initiatives  for  maintenance 
planners. 

340 

238 

70.0 

Table  1  continued  on  next  page 


Table  1  continued 


Course  Name 

Course 

Number 

Course  Description 

Number 

Enrolled 

Number 

Completed 

E'er 

Pricing  Analysis  Methods 

QMTllO 

A  concentrated  course  on  price  analysis  methods. 

419 

294 

70.2 

Introduction  to 
Evolutionary  Acquisition 

SYS  149 

Provides  guidance  on  using  evolutionary  acquisition 
as  strategy  for  achieving  shorter  acquisition  cycles. 

439 

312 

71.1 

Reducing  Acquisition 
response  Time 

SYS350 

Presents  impact  of  long  development  times  on  war¬ 
fighter,  budget,  acquisition,  and  sustainment 
communities. 

215 

153 

71.2 

Contract  Repair  Process 

SYS  175 

Presents  contract  repair  process  used  by  logistics 
commands. 

309 

221 

71.5 

Integrated  Product 

Support  Course 

SYS171 

Provides  student  with  basic  processes  supporting 
major  systems. 

788 

565 

71.7 

AFRL  R&D  Case  File 
Management  Course 

FAB  150 

Covers  procedures  for  maintaining  good 
documentation  while  ensuring  continuity  is 
maintained  between  researchers. 

966 

693 

71.7 

Product  Support 
Management  Planning 

SYS  173 

Covers  support  requirements,  sources  of  support 
migration  planning,  partnering  policy  and  service 
level  agreements. 

199 

144 

72.4 

Modification 

Management 

SYS  172 

Emphasizes  the  reengineered  modification 
management  process. 

583 

430 

73.8 

Commercial  Business 
Approach 

SYS  195 

Covers  commercial  business  approach  to  acquisition 
programs. 

349 

270 

77.4 

Operational  Safety. 
Suitability  and 
Effectiveness  (OSS&E) 

SYS155 

Course  enables  Air  Force  Material  Command  to 
begin  policy  execution  to  institutionalize  OSS&E 
processes. 

750 

588 

78.4 

Measurements 


Course  Completion  Rates 

Course  completion  rates  refer  to  the  number  of  students  who  complete  an  e- 
learning  course  compared  to  the  number  who  enrolled.  Completion  rates  were  computed 
from  statistics  maintained  on  the  VSH  website  using  the  following  formula: 

Ecr  =  total  students  completed 
total  students  enrolled 

Course  Dijficulty 

For  purposes  of  this  study,  course  difficulty  refers  to  how  hard  e-learning  courses 
are  in  terms  of  their  Match  Between  System  and  the  Real  World  heuristic  score  from  the 
Heuristic  Evaluation  Checklist,  readability,  and  grade  level.  The  heuristic  is  described  in 
the  Instrument  Development  section  of  this  chapter.  Readability  is  based  on  the  Flesch 
Reading  Ease  Score,  which  is  determined  by  the  structure  of  words  and  sentences 
(Elesch,  1962).  Its  scale  ranges  from  1  to  100.  The  higher  the  score,  the  easier  the  text  is 
to  read.  To  collect  this  data,  each  screen  of  text  in  the  courses  was  copied  to  Microsoft 
Word.  Then  the  Microsoft  Word  readability  statistics  tool  was  turned  on  and  run  against 
the  text.  The  readability  scores  were  then  averaged  to  come  up  with  an  overall  readability 
score  for  each  course. 

Grade  level  is  based  on  the  Flesch-Kincaid  Grade  Eevel  score  and  is  determined 
by  converting  the  readability  score  to  a  United  States  grade  level.  This  data  was  collected 
and  computed  in  the  same  way  as  the  readability  score. 

Reynolds  (2002)  used  a  similar  construct  to  determine  a  relationship  between 
course  difficulty  and  course  completion  rates.  Based  on  a  sample  of  five  VSH  courses. 
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Reynolds  found  no  statistically  reliable  relationship  between  course  difficulty  and 
completion  rates.  This  study  will  retest  the  difficulty  concept  using  a  larger  sample  of 
courses  to  determine  if  sample  size  influenced  Reynolds’  results.  Table  (2)  shows  the 
descriptive  characteristics  of  the  difficulty  construct  computed  across  courses.  See 
Appendix  B  for  individual  course  results. 


Table  2,  Descriptive  Characteristics  for  Course  Difficulty 


Characteristic 

M 

SD 

Questions  Stated  in  Clear  Simple  Language, 

4.74 

0.56 

Readability  (Flesch  Reading  Ease  Score) 

31.28 

8.28 

Grade  Level  (Flesch-Kincaid) 

11.27 

0.63 

Note.  Bold  text  indicates  item  included  in  computation  of  Pearson  Correlation. 


Support  for  Hypothesis  2  {H2)  will  be  based  on  the  statistically  reliable 
relationship  between  each  the  course  difficulty  factors  and  course  completion  rates.  The 
heuristic  evaluation  score  and  readability  must  be  positively  correlated  and  the  grade 
level  must  be  negatively  correlated.  There  will  be  no  support  for  H2  if  none  of  the 
relationships  is  statistically  reliable  as  predicted.  One  or  two  statistically  reliable 
relationships  will  be  considered  partial  support  for  H2.  Reliable  relationships  between  all 
three  course  difficulty  factors  and  course  completion  rates  will  indicate  strong  support  for 
H2. 
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Words  per  Module  and  Broken/Misdirected  Hyperlinks 

To  find  the  number  of  words  per  module,  the  text  on  eaeh  sereen  is  eopied  into 
Mierosoft  Word  and  the  word  eount  tool  run.  The  total  words  for  eaeh  sereen  are  summed 
and  then  divided  by  the  number  of  modules  in  the  eourse  to  find  an  average  module 
length  for  eaeh  eourse  (see  Appendix  B).  These  measures  are  used  to  test  Hg. 

The  number  of  broken  and  misdireeted  hyperlinks  was  determined  by  following 
all  hyperlinks  in  a  eourse  and  reeording  the  links  that  do  not  eonneet  to  the  information 
they  represented.  This  measure  is  used  to  test  Hg. 

Table  3  shows  the  deseriptive  statisties  of  the  measurements  for  these  two 
heuristies  eomputed  aeross  eourses.  Both  items  were  ineluded  in  the  Pearson  Correlation 
analysis. 


Table  3,  Descriptive  Statistics  for  Module  Length  and  Broken  Hyperlinks 


Characteristic 

M 

SD 

Module  Length 

2137.75 

1182.40 

Broken  Links  (%) 

0.16 

0.13 

Note.  Bold  text  indieates  item  ineluded  in  eomputation  of  Pearson  Correlation. 


Heuristic  Evaluation  Checklist 

A  review  of  eurrent  literature  on  heuristie  evaluation  yielded  an  existing  graphieal 
user  interfaee  evaluation  tool,  ealled  “Heuristic  Evaluation  -  A  System  Checklist” 
(Pierotti,  2002),  that  could  be  easily  tailored  and  adapted  for  use  in  evaluating  e-learning 
course  designs.  Based  on  Nielsen’s  (1994)  heuristics,  the  checklist  provides  a  set  of 
design  characteristics  for  each  heuristic  that  can  be  tailored  to  evaluate  the  VSH  courses. 
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Rather  than  use  the  3  point  “Yes-No-N/A”  seale  used  by  Pierotti,  this  study  implemented 
a  5  point  Likert  scale  ranging  from  Almost  Never  to  Almost  Always.  This  scale  was 
incorporated  to  give  the  evaluator  more  discretion  in  assessing  how  well  a  course 
demonstrated  compliance  with  a  characteristic.  The  following  sections  describe  each  of 
the  heuristic’s  characteristics  chosen  to  evaluate  the  VSH  courses  along  with  the  mean 
and  standard  deviation  computed  across  the  courses  from  the  evaluation  results. 


Visibility  of  System  Status 

The  first  heuristic,  Visibility  of  System  Status,  was  used  to  evaluate  whether  the 
system  kept  the  user  informed  about  what  is  going  on,  through  appropriate  feedback 
within  reasonable  time  (Nielsen,  1994).  Table  4  summarizes  the  descriptive  statistics  for 
the  visibility  of  system  status  heuristic. 


Table  4,  Descriptive  Statistics  for  Visibility  of  System  Status 


Characteristic 

M 

SD 

Title  or  Header  on  Every  Page 

5.00 

0.00 

Pop-up  Windows  Cover  Fields 

4.57 

1.13 

System  Feedback  for  Every  Operator  Action 

4.79 

0.71 

Indication  to  Start  Next  Group  of  Actions 

5.00 

0.00 

Feedback  on  Where  to  Put  Moved  Objects 

4.33 

1.21 

User  Informed  During  System  Delays 

2.00 

— 

Place  Markers  as  Navigational  Aids 

1.10 

0.45 

Note.  Bold  text  indicates  item  included  in  computation  of  Pearson  Correlation. 
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Analysis  of  the  descriptive  statistics  revealed  that  some  characteristics  would  not 
be  useful  in  computing  the  Pearson  Correlation  and  should  be  excluded.  Title  or  header 
on  every  screen  and  indication  to  start  next  group  of  actions  were  excluded  because  there 
was  no  variation.  All  of  the  courses  fully  implemented  these  characteristics.  Pop-up 
windows  cover  fields  was  removed  because  it  only  applied  to  seven  courses  and  only  one 
course  score  differed  from  the  rest.  User  informed  during  system  delays  was  removed 
because  only  one  course  experienced  a  delay.  Place  markers  as  navigational  aids  was 
removed  because  only  one  VSH  course  used  place  markers. 

Match  Between  System  and  the  Real  World 

The  heuristic  of  Match  Between  System  and  the  Real  World  was  used  to  evaluate 
whether  the  system  spoke  the  user’s  language,  with  words,  phrases  and  concepts  familiar 
to  the  user,  rather  than  system  oriented  terms  (Nielsen,  1994).  See  Table  2  for  the 
distinguishing  characteristics  for  this  heuristic. 

User  Control  and  Freedom 

The  heuristic  of  User  Control  and  Freedom  was  used  to  evaluate  whether  users 
could  easily  recover  from  unintended  actions  without  having  to  go  through  an  extended 
dialogue  (Nielsen,  1994).  Table  5,  on  the  next  page,  shows  the  descriptive  statistics  for 
this  heuristic.  Analysis  of  the  descriptive  statistics  for  rearranging  windows,  waiting  for  a 
signal  from  the  user,  and  scroll  bars  resulted  in  exclusion  of  these  three  items  from 
further  analysis  due  to  lack  of  variation  in  the  scores. 
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Table  5.  Descriptive  Statistics  for  User  Control  and  Freedom 


Characteristic  M  SD 

Easy  to  Rearrange  Windows  on  Screen  5.00  0.00 

Easy  to  Switch  Between  Windows  4,84  0,50 

System  Waits  for  Signal  From  User  Before  Processing  4.95  0.22 

Mechanism  Allowing  Users  to  Go  Back  to  Previous  4,20  1,01 

Menus 

Users  Can  Go  Forward  and  Back  Through  Questions  1,80  1,10 

Vertical  and  Horizontal  Scroll  Bars  Provided  5.00  0.00 


Note.  Bold  text  indicates  item  included  in  computation  of  Pearson  Correlation. 
Consistency  and  Standards 

The  heuristic  for  consistency  and  standards  was  used  to  evaluate  whether  users 
had  to  wonder  if  different  words,  situations,  or  actions  mean  the  same  thing.  Table  6 
shows  the  descriptive  statistics  for  the  characteristics  evaluated  for  this  heuristic. 


Table  6,  Descriptive  Statistics  for  Consistency  and  Standards 


Characteristic 

M 

SD 

Consistent  Formatting  Standards  Followed 

4,25 

1,12 

Heavy  Use  of  All  Uppercase  Letters  Avoided 

5.00 

0.00 

Icons  Labeled 

4,70 

0,73 

Attention-Getting  Techniques  Used  With  Care 

4.05 

0,89 

Valid  Inputs  for  Question  Listed 

4,75 

0,71 

Note.  Bold  text  indicates  item  included  in  computation  of  Pearson  Correlation. 
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Analysis  of  the  descriptive  statistics  for  this  heuristic  revealed  no  variation  in  the 
heavy  use  of  uppercase  letters  so  this  item  was  excluded  from  further  analysis. 

Aesthetic  and  Minimalist  Design 

Nielsen  (1994)  stated  in  the  Aesthetic  and  Minimalist  Design  heuristic  that 
dialogues  should  not  contain  information  which  is  irrelevant  or  rarely  needed.  Table  7 
shows  descriptive  statistics  for  the  characteristics  evaluated  for  this  heuristic.  All  of  the 
characteristics  were  used  in  the  correlation  analysis. 


Table  7.  Descriptive  Statistics  for  Aesthetic  and  Minimalist  Design 


Characteristic 

M 

SD 

Information  Essential  to  Decision-Making 

4.30 

0.98 

Brief  Menu  Titles 

4.95 

.22 

Color  Used  With  Discretion 

3.95 

1.00 

Note.  Bold  text  indicates  item  included  in  computation  of  Pearson  Correlation. 


Help  and  Documentation 

Nielsen  (1994)  further  stated  in  the  Help  and  Documentation  heuristic  that  though 
it  would  be  better  if  the  system  can  be  used  without  help  and  documentation,  it  may  be 
necessary  to  provide  it.  Any  such  information  should  be  easy  to  search,  focused  on  the 
task,  and  not  be  too  large.  Table  8,  on  the  next  page,  shows  the  descriptive  statistics  of 
the  characteristics  used  to  evaluate  the  help  provided  in  each  course. 
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Table  8,  Descriptive  Statistics  for  Help  and  Documentation 


Characteristic 

M 

SD 

Help  Function  Visible 

3.10 

1.86 

Information  Relevant 

2.6 

0.99 

Easy  to  Access  and  Return  from  Help 

2.75 

1.92 

Errors  in  Matching  Exercises  Using  Connecting  Eines 

2.00 

— 

Note.  Bold  text  indicates  item  included  in  computation  of  Pearson  Correlation. 


Skills 

This  heuristic,  as  summarized  by  Pierotti  (2002)  dictates  that  the  system  should 
support  a  user’s  computer  skills,  not  replace  them.  Table  9  shows  the  descriptive  statistics 
for  the  characteristics  used  to  evaluate  how  well  the  courses  supported  the  user’s 
computer  skills. 


Table  9,  Descriptive  Statistics  for  Support  of  User’s  Skills 


Characteristic 

M 

SD 

Window  Operations  Easy  to  Learn  and  Use 

3.90 

1.21 

Support  for  Novice  and  Expert  Users 

— 

— 

Cursor  Positioned  in  Textbox 

1.60 

0.55 

Note.  Bold  text  indicates  item  included  in  computation  of  Pearson  Correlation. 


Support  for  novice  and  expert  users  looked  at  things  like  using  shortcut  keys; 
hidden  commands  that  someone  familiar  with  the  system  could  use  to  use  the  system 
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faster.  This  characteristic  was  not  observed  within  any  the  courses.  It  seemed  like  a 
logical  item  when  it  was  selected  for  the  checklist.  The  cursor  position  item  was  excluded 
from  further  analysis  because  it  only  applied  to  a  few  courses  and  there  was  very  little 
variation  of  scores  where  it  did  apply. 

Statistical  Analysis 

The  statistical  technique  employed  in  this  study  was  Pearson’s  correlation.  Under 
the  assumption  of  normality,  a  Pearson  correlation  is  used  to  determine  the  linear 
relationship  between  a  pair  of  variables.  SPSS  was  used  to  create  a  correlation  matrix 
showing  the  relationships  between  the  criterion  variable,  course  completion  rates,  and  the 
predictor  variables:  module  length,  broken  links,  readability,  grade  level,  and,  for  each 
heuristic,  the  mean  of  each  characteristics  for  each  course. 

All  statistical  analyses  are  reported  in  Chapter  IV.  The  goal  was  to  find 
statistically  reliable  relationships  to  asses  the  hypotheses  presented  in  Chapter  II. 

Summary 

This  chapter  explained  the  research  approach  and  methodology  used  to  identify 
course  design  factors  that  influence  e-learning  course  completion  rates.  Pearson’s 
correlation  was  used  to  identify  any  statistically  reliable  relationships  between 
completion  rates  (^cr)  and  data  collected  from  an  evaluation  of  VSH  courses  using  the 
Heuristic  Evaluation  Checklist,  and  other  measures.  The  result  will  then  be  used  to  draw 
conclusions  about  the  effects  design  factors  have  on  e-learning  course  completion  rates. 
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IV.  Results 


“One  should  never  spoil  a  good  theory  by  explaining  it.  ” 

Peter  McArthur 

Introduction 

The  intent  of  this  study  was  to  identify  e-leaming  course  design  factors  that 
influence  course  completion  rates.  This  chapter  presents  the  analysis  and  findings  from 
the  Heuristic  Evaluation  Checklist,  as  well  as  other  measurements  taken  to  support  or 
refute  the  nine  hypotheses  presented  in  chapter  two.  First,  a  table  is  presented  that 
summarizes  descriptive  statistics  and  the  linear  relationships  of  each  variable  to  VSH 
course  completion  rates.  The  table  is  then  followed  by  a  discussion  of  each  variable 
correlation. 

Descriptive  Statistics 

Table  10  shows  the  mean,  standard  deviation,  and  the  Pearson  correlation  for  the 
study  variables.  Because  the  focus  of  this  study  was  limited  to  the  effects  of  design 
factors  on  e-learning  course  completion  rates,  discussion  will  be  limited  to  the 
relationships  between  completion  rates  and  the  other  study  variables. 

Unsupported  Hypotheses 

As  Table  10  on  the  next  page  shows,  the  correlations  between  course  completion 
rates  and  visibility  of  system  status,  consistency  and  standards,  aesthetic  and  minimalist 
design,  and  broken  links  were  not  statistically  reliable.  Therefore,  hypotheses  Hi,  H4,  H5, 
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and  Hg  are  not  supported.  Additionally,  none  of  the  correlations  for  the  variables 
measuring  course  difficulty  (match  between  system  and  real  world,  readability,  grade 
level)  were  statistically  reliable;  therefore,  H2  is  not  supported.  Although  statistically 


Table  10,  Descriptive  Statistics  and  Correlation  Matrix 


Variable 

Mean 

SD 

Completion  Rate 
Correlation 

Visibility  of  System  Status 

4.78 

0.70 

-.25 

Match  Between  System  and  Real  World 

4.74 

0.56 

-.11 

User  Control  and  Freedom 

4.28 

0.83 

.31  * 

Consistency  and  Standards 

4.51 

0.45 

.26 

Aesthetic  and  Minimalist  Design 

4.40 

0.55 

-.03 

Help  and  Documentation 

2.82 

1.46 

.31  * 

Skills 

3.90 

1.21 

-.34  * 

Words  per  Module 

1914.00 

628.05 

-.51  * 

Readability 

31.28 

8.07 

-.13 

Grade  Level 

11.27 

0.61 

-.19 

Broken  Links 

.16 

0.12 

.13 

Note.  An  *  indicates  one-tail  correlations  statistically  reliable  at/>  <  .10.  Sample  size 
for  all  correlations  except  words  per  module  was  A=  20.  Sample  size  for  words  per 
module  was  N=  19. 
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reliable,  the  inverse  relationship  between  course  completion  rates  and  skills  was  not  as 
predicted.  Therefore,  Hj  is  not  supported. 

Supported  Hypotheses 

The  following  paragraphs  discuss  those  hypotheses  that  were  supported  in  this 
study.  The  supported  hypotheses  were  H3,  Hs,  and  Hs. 


H3  -  User  Control  and  Freedom 

The  correlation  between  course  completion  rates  and  factors  influencing  user 
control  and  freedom  was  statistically  significant  (r  =  3\,p<  .10).  Figure  2  shows  the 


Figure  2,  Scatter  Diagram  for  User  Control  and  Freedom, 
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scatter  diagram  depicting  the  positive  relationship  between  the  observed  user  control  and 
freedom  factors  and  course  completion  rates.  These  findings  suggest  that  users  who  are 
able  to  exercise  greater  control  over  the  e-leaming  environment,  and  have  freedom  to 
easily  move  back  and  forth  through  course  materials,  are  more  likely  to  complete  the  e- 
learning  course.  Therefore,  H3  is  supported. 

//fi  -  Help  and  Documentation 

The  correlation  between  course  completion  rates  and  the  level  and  quality  of  the 
help  and  documentation  within  a  VSH  e-learning  course  was  found  to  be  statistically 
significant  (r  =  .31,/>  <  .10).  Figure  3  shows  the  scatter  diagram  depicting  the  positive 


Figure  3,  Scatter  Diagram  for  Help  and  Documentation 
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relationship  between  the  observed  complianee  of  help  and  documentation  characteristics 
and  course  completion  rates.  These  findings  suggest  that  a  help  system  that  is  easily 
accessible  and  relevant  to  the  user’s  task  can  assist  a  user  in  completing  an  e-learning 
course.  Therefore,  //g  is  supported. 

Use  of  Chunking 

Applying  the  concept  of  chunking  to  a  VSH  e-leaming  course  means  to  break  the 
course  into  manageable  pieces,  or  modules  that  should  allow  the  user  to  complete  a 
module  before  off-task  demands  interfere  with  completion  of  the  module  and,  therefore, 
completion  of  the  course.  It  was  expected  that  the  correlation  between  course  completion 
rates  and  word  per  module  would  be  inverse;  that  is,  as  the  number  of  words  per  module 
increase,  course  completion  rates  decrease.  The  correlation  between  course  completion 
rates  and  average  module  length  for  all  twenty  VSH  courses  was  not  statistically 
significant  across  all  twenty  courses  (r  =  -0.\1  ,p>  .10).  Examination  of  the  scatter 
diagram  shown  in  Figure  4  on  the  next  page  revealed  that  the  measurement  for  one 
course.  Introduction  to  Evolutionary  Acquisition  (EA),  appeared  very  different  from  the 
other  courses.  The  EA  course  only  contained  2  modules  while  the  average  across  the 
other  nineteen  courses  was  8.2  modules.  One  of  the  modules  in  the  EA  course  was  an 
interactive  exercise  and  none  of  the  other  courses  had  anything  like  this.  I  felt  these 
differences  were  sufficient  to  classify  the  data  point  as  an  outlier  and  warranted  removal 
from  the  data  analysis.  After  removing  the  data  related  to  the  Evolutionary  Acquisition 
course,  the  correlation  between  course  completion  rates  and  words  per  module  becomes 
statistically  significant  (r  =  -.51  .05),  indicating  strong  support  for///. 
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Words  per  Module 

Figure  4,  Analysis  of  Words  per  Module 

Summary 

Table  1 1  on  the  next  page  eontains  a  summary  of  the  hypothesized  relationships 
between  the  heuristies  and  eourse  eompletion  rates  tested  in  this  study.  Three  of  the  nine 
hypotheses  were  supported.  The  results  provide  some  support  for  using  heuristic 
evaluation  to  identify  course  design  characteristics  that  influence  course  completion  rates. 
These  results  are  discussed  further  in  the  final  chapter. 
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Table  11.  Summary  of  Hypotheses 

Hypotheses 

Hi’,  a  positive  relationship  for  the  visibility  of  system  status 
Hr.  an  inverse  relationship  for  the  diffieulty  of  the  eourse  material 
Hs’.  a  positive  relationship  for  the  level  of  user  eontrol  and  freedom 
Hr  a  positive  relationship  for  the  level  of  eomplianee  with  platform 
eonsisteney  and  standards 

Hs’.  a  positive  relationship  for  aesthetie  and  minimalist  design 
Hr  a  positive  relationship  for  the  level  of  help  and  doeumentation 
Hj’.  a  positive  relationship  for  the  level  of  support  for  the  user’s  skills 
Hs’.  an  inverse  relationship  for  the  average  length  of  the  modules 
Hr  an  inverse  relationship  for  the  pereentage  of  broken  hyperlinks 


Supported 

No 

No 

Yes 

No 

No 

Yes 

No 

Yes 

No 
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V.  Discussion 


“All  words  are  pegs  to  hang  ideas  on.  ” 

Henry  Ward  Beecher 

Introduction 

Twenty  VSH  courses  were  analyzed  using  the  usability  inspection  method  of 
heuristic  evaluation.  A  heuristic  is  a  recognized  usability  principle  that  guides  the 
evaluator  as  they  attempt  to  identify  usability  problems  within  an  interface  (Nielsen, 
1994:26).  Nine  heuristics  were  chosen  for  this  study  that  formed  the  basis  for  the  nine 
hypotheses  used  to  address  the  following  research  question: 

Can  design  factors  be  identified  that  influence  e-learning  course  completion 

rates? 

This  chapter  begins  with  a  discussion  of  the  observations  associated  with  the  each 
of  the  nine  hypotheses.  Next,  implications  for  the  study  are  addressed,  followed  by 
discussion  of  limitations  and  recommendations  for  future  research. 

Observations 

Results  of  the  study  indicate  some  support  for  using  the  heuristic  evaluation 
method  to  identify  design  factors  that  influence  course  completion  rates.  The  following 
sections  discuss  the  heuristics  associated  with  each  hypothesis  and  the  specific 
observations  about  why  the  different  characteristics  were  scored  as  they  were. 

The  discussions  also  include  recommendations  for  exclusion  of  certain  characteristics 
where  deemed  appropriate. 
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Visibility  of  System  Status 

The  first  hypothesis  looked  for  a  relationship  between  course  completion  rates 
and  how  well  the  VSH  system  kept  the  user  informed  of  system  status.  There  were  four 
characteristics  of  system  status  evaluated. 


Title  or  Header  on  Every  Screen.  All  of  the  courses  in  the  Virtual  Schoolhouse 
fully  implemented  this  characteristic.  All  of  the  screens  contained  a  title  or  header  that 
informed  the  student  of  the  subject  being  discussed.  I  found  this  feature  helpful  when 
trying  to  review  the  information  during  an  exercise.  Recommend  keeping  this  inspection 
item  in  future  research. 


Pop-up  Windows  Cover  Fields.  There  was  only  one  course  that  exhibited  a 
discrepancy  for  this  characteristic;  however,  it  was  very  annoying  when  it  happened.  The 
one  example  of  this  was  on  the  Key  Steps  page  of  the  Performance  Based  Payments 
module,  in  the  Pricing  Analysis  Methods  course  where  the  student  had  to  place  the  mouse 
cursor  over  the  text  line  to  make  more  text  appear.  If  the  mouse  was  placed  over  the 
number,  then  the  student  could  read  the  text  even  though  the  alternate  text  tag  appeared.. 
But  if  the  mouse  was  placed  over  the  text,  the  alternate  text  tag  would  obscure  the 
underlying  text  and  the  student  could  not  read  it. 


System  Feedback  for  Every  Operator  Action.  This  item  was  scored  low  where  the 
course  provided  no  feedback  for  many  review  questions.  In  other  words,  the  student  isn’t 
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told  whether  they  were  right  or  wrong.  The  low  seoring  course  on  this  characteristic  was 
the  Contract  Repair  Process  course.  The  system  did  not  provide  feedback  to  the  student 
on  whether  their  answer  to  many  of  the  review  questions  was  correct  or  wrong.  It 
appeared  that  a  necessary  segment  of  JavaScript  code  was  missing  as  some  question  did 
provide  feedback  and  there  was  code  on  the  page  to  do  that.  A  medium  score  indicated 
just  a  few  questions  provided  no  feedback.  The  Commercial  business  Approach  course 
had  a  couple  of  questions  where  no  feedback  was  provided.  High  scores  indicated  good 
feedback  on  all  questions.  Most  of  the  courses  received  a  high  score. 

Indication  to  Start  Next  Group  of  Actions.  This  item  looked  at  how  the  student 
knew  they  were  finished  with  a  module  and  ready  to  begin  the  next  one.  The  student  was 
kept  informed  of  this  information  on  the  class  listing  page  with  a  complete  or  incomplete 
status  value,  and  on  the  course  module  page  with  a  pass  or  incomplete  status  value.  All 
courses  reported  this  information  to  the  student. 

Feedback  On  Where  to  Put  Moved  Objects.  Some  of  the  courses  had  exercises 
that  required  the  student  to  click  on  an  object,  then  drag  and  drop  it  somewhere  else  on 
the  screen.  Some  of  these  exercises  did  not  have  an  indication  of  exactly  where  the  object 
should  be  positioned  when  the  user  dropped  it.  The  object  would  then  jump  to  another 
location  on  the  screen,  forcing  the  user  to  keep  trying  until  finally  locating  the  exact 
position  that  was  needed.  This  made  performing  these  exercises  difficult  and  time 
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consuming.  Courses  receiving  low  scores  on  this  characteristic  contained  exercises  that 
displayed  this  difficulty. 

User  Informed  During  System  Delays.  There  were  no  system  delays  noted  during 
evaluation  of  the  courses. 

Place  Markers  as  Navigational  Aids.  For  the  purposes  of  this  study,  a  place 
marker  was  considered  to  be  an  indicator  on  the  screen  of  the  number  of  pages  within  a 
module,  and  an  indication  of  which  page  the  user  was  currently  on.  Only  one  course 
implemented  this  type  of  mechanism.  It  was  noted  by  the  evaluator  a  number  of  times 
that  this  would  have  been  very  useful  information  for  deciding  whether  to  continue 
working  on  a  module  or  to  stop  to  take  care  of  some  off-task  demands.  Recommend 
adding  this  navigation  feature  to  all  VSH  courses. 

Course  Difficulty 

The  second  hypothesis  looked  for  a  relationship  between  course  completion  rates 
and  course  difficulty.  A  combination  of  measurements  was  used  to  test  this  hypothesis; 
the  one  characteristic,  dealing  with  the  clarity  of  language  in  review  questions  under  the 
Match  Between  System  and  the  Real  World  heuristic,  readability,  and  grade  level. 

Questions  Stated  in  Clear  Simple  Language.  This  item  evaluated  how  clearly 
review  questions  were  stated.  Most  of  the  courses  scored  high  on  this  characteristic.  An 
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example  of  a  medium  score  is  the  Modification  Management  course.  This  course  used  a 
lot  of  acronyms  in  a  number  of  questions  that  the  student  may  not  have  memorized.  This 
forced  a  lot  of  searching  back  through  the  document  just  to  get  familiar  with  the 
acronyms.  Perhaps  access  to  an  acronym  list  on  every  page  would  alleviate  this  issue. 

Readability.  Readability  was  measured  using  the  Flesch  Reading  Ease  score 
(Flesch,  1962).  Though  the  mean  readability  across  all  of  the  courses  (31.28)  is  in  the 
difficult  to  read  range  (Flesch,  2003),  this  characteristic  was  not  statistically  significant 
and  was,  in  fact,  inversely  related  to  completion  rates.  The  Army  has  used  this  method 
extensively  in  testing  training  manuals  and  found  it  insufficient  for  matching  up  the 
reading  skills  of  soldiers  to  their  ability  to  understand  the  content  (Kern,  1980).  Other 
studies  (Reynolds,  2002)  have  also  found  no  support  for  using  this  tool  to  measure  e- 
leaming  course  difficulty.  I  would  recommend  not  using  this  tool,  or  this  measure,  in 
future  research. 

Grade  Level.  Grade  level  was  determined  using  the  Flesch-Kincaid  Grade  Fevel 
tool  provided  in  Microsoft  Word.  The  formula  for  finding  the  Flesch-Kincaid  Grade  level 
converts  the  reading  ease  score  to  a  U.S.  grade  level.  The  mean  grade  level  across  all 
courses  in  VSH  was  1 1.27,  which  means  the  reader  should  have  an  1 1*’’  grade  education 
to  understand  the  material.  However,  examination  of  the  data  revealed  a  large  number  of 
scores  equaling  12.  It  was  discovered  after  collection  of  the  data  that  the  grade  level  tool 
provided  in  Microsoft  Word  reports  a  maximum  score  of  12  (Wats.ca,  2003).  This  could 
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mean  that  the  grade  level  of  the  material  was  aetually  higher  than  reported  for  this  study. 
For  this  reason,  and  beeause  the  grade  level  is  a  linear  transformation  of  the  readability 
seore,  I  would  reeommend  not  ineluding  this  measure  in  future  researeh. 

User  Control  and  Freedom 

The  third  hypothesis  looked  for  a  relationship  between  course  completion  rates 
and  user  control  and  freedom  factors.  There  were  six  characteristics  of  user  control  and 
freedom  evaluated. 

Easy  to  Rearrange  Windows  On  The  Screen.  The  functionality  behind  this 
characteristic  was  actually  provided  by  the  web  browser  and  not  through  any 
functionality  provided  within  the  courses.  For  this  reason,  I  recommend  excluding  this 
characteristic  from  future  design  research  on  the  VSH  system. 

Easy  to  Switch  Between  Windows.  This  characteristic  evaluated  how  easy  it  was 
for  users  to  return  to  a  page  that  a  new  window  had  been  launched  from.  While  there 
were  no  low  scores  for  this  characteristic,  a  medium  score  was  given  for  the  two  courses 
that  failed  to  provide  a  close  mechanism  within  the  text  of  the  window.  Not  having  a 
readily  available  method  for  closing  the  window  forces  the  user  to  spend  time  searching 
for  the  window  close  icon  provided  by  the  operating  system.  Even  though  few  courses 
were  discrepant  on  this  characteristic,  it  is  a  convenience  that  makes  course  navigation 
easier.  I  would  recommend  having  a  close  feature  on  all  pop-up  screens  within  the  VSH. 
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System  Waits  for  Signal  From  User  Before  Processing.  Evaluation  for  this 
characteristic  looked  for  instanees  where  the  system  would  automatieally  begin 
proeessing  some  aetion  without  the  user  initiating  it.  None  of  the  eourses  displayed  a 
problem  in  this  area. 

Mechanism  Allowing  Users  to  Go  Back  to  Previous  Menu.  Most  of  the  eourses 
provided  an  ieon  on  the  sereen  that  allowed  the  user  to  return  to  the  module  menu  page  at 
the  beginning  of  the  eourse.  Those  eourses  seoring  low  on  this  eharaeteristie  only 
provided  navigation  meehanisms  for  moving  ahead  to  the  next  page  and  back  to  the 
previous  page.  The  ability  to  navigate  baek  to  the  module  menu  page  made  it  easier  to 
jump  from  one  module  to  another  to  review  information.  Reeommend  providing  the 
eapability  to  navigate  to  a  module’s  menu  page  on  all  sereens  within  a  eourse. 

Users  Can  Go  Forward  and  Back  Through  Questions.  After  a  few  eourse 
evaluations  had  been  completed,  it  was  determined  that  this  eharaeteristie  was  not  really 
applieable  to  the  VSH.  One  of  the  objeetives  for  the  VSH  eourses  is  that  the  students 
eorrectly  answer  all  of  the  review  questions,  and  moving  baek  and  forth  between  the 
review  questions  was  not  something  that  needed  to  be  done.  Reeommend  removing  this 
item  from  future  evaluations  of  the  VSH  system. 

Vertical  and  Horizontal  Scroll  Bars  Provided.  It  is  important  to  ensure  that  seroll 
bars  are  available  when  eourses  provide  information  to  the  student  in  the  form  of  a  pop¬ 
up  window;  espeeially  if  the  window  eannot  be  resized  to  fit  the  eontents.  There  was  only 
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one  instance  where  this  problem  occurred  but  I  was  unable  to  recreate  the  problem  a 
second  time.  Therefore,  I’m  not  sure  it  was  a  problem  with  the  course  or  a  problem  with 
the  computer.  Either  way,  navigation  of  the  material  was  impossible.  To  ensure  that  this 
was  not  a  problem  with  the  course,  I  recommend  verifying  that  all  pop-up  windows 
provide  scroll  bars  whenever  the  VSH  courses  are  being  evaluated. 

Consistency  and  Standards 

The  fourth  hypothesis  looked  for  a  relationship  between  course  completion  rates 
and  design  factors  for  consistency  and  standards  on  a  web  page.  There  were  five 
characteristics  of  consistency  and  standards  evaluated. 

Consistent  Formatting  Standards  Followed.  Low  scores  on  this  characteristic 
were  given  on  courses  that  used  underlined  text  that  was  not  a  hyperlink.  The  convention 
for  web  pages  is  that  underlined  text  indicates  a  hyperlink  that  will  provide  additional 
information  related  to  the  underlined  text.  This  causes  an  unnecessary  distraction  for 
students  when  there  are  other  ways  of  highlighting  text  on  a  page.  Recommend  not  using 
underlining  on  VSH  course  screens.  A  course  received  a  medium  score  on  this 
characteristic  if  the  layout  of  the  screen  was  significantly  different  across  pages  in  a 
course.  As  the  student  uses  the  course,  he/she  can  become  familiar  with  the  location  on 
the  screen  where  text  will  appear  next.  This  helps  decrease  the  time  needed  to  locate  text 
and  begin  reading.  I  have  no  recommendation  for  where  to  locate  the  text  and  graphics  on 
the  screen,  just  that  it  be  consistently  located  in  the  same  area  on  the  screen. 
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Heavy  Use  of  All  Uppercase  Letters  Avoided.  It  is  web  convention  that  the  use  of 
all  uppercase  letters  indicates  the  sender  of  the  text  is  yelling  at  the  reader.  Use  of  all 
uppercase  letters  was  not  observed  in  any  of  the  courses. 

Icons  Labeled.  Icons  provided  within  a  VSH  course  may  be  unfamiliar  and  should 
therefore  have  some  indication  of  their  function  visible  to  the  user.  A  picture  or  other 
graphic  may  not  be  easily  recognized  so  a  label  of  some  kind  may  be  necessary  to  inform 
the  user  of  an  icon’s  function.  The  Price  Analysis  Methods  course  was  the  only  course 
that  used  icons,  specifically,  navigation  icons  that  were  not  labeled.  The  function  of  these 
icons  was  easily  recognized  and  did  not  present  a  problem  with  using  the  VSH  system. 

Attention-Getting  Techniques  Used  With  Care.  An  attention-getting  technique  can 
be  a  distraction  in  a  course  when  the  technique  does  not  enhance  the  learning  experience. 
The  VSH  course  that  scored  lowest  on  this  characteristic  was  Intelligence  in  Force 
Modernization.  This  course  had  flaming  matchsticks  beside  the  menu  items  on  the 
module  menu  pages.  This  feature  appeared  to  be  just  another  distraction  in  the  course. 

VSH  courses  that  received  a  medium  score  on  this  characteristic  used  a  number  of 
attention-getting  techniques  such  as  spinning  saw  blades,  spinning  gears,  and  other 
moving  objects  that  did  not  enhance  the  learning  experience.  While  it  is  important  to  try 
to  make  the  courses  interesting,  this  type  of  content  competes  for  the  attention  of  the 
student  with  the  educational  material  and  distracts  them  from  their  task.  Recommend 
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VSH  course  designers  and  administrators  limit  the  use  of  these  attention-getting 
teehniques  to  those  that  eontribute  to  understanding  the  material. 

Valid  Inputs  For  a  Question  Listed.  The  Earned  Value  Management  System 
eourse  reeeived  the  only  low  seore  on  this  eharaoteristie.  The  problem  observed  was  that 
a  review  question  would  require  the  user  to  type  in  an  answer  in  a  eertain  format.  If  the 
answer  was  eorreet,  but  in  the  wrong  format,  the  student  would  reeeive  feedbaek  that 
he/she  had  answered  ineorreetly.  An  example  of  this  is  a  question  that  required  the 
answer  “SOW”.  SOW  is  an  aeronym  for  statement  of  work,  but  if  the  student  typed  in 
“Statement  of  Work”,  the  feedbaek  indieated  an  ineorrect  answer.  The  student  diseovered 
that  the  aeronym  was  required  only  after  seeing  the  feedbaek  about  the  ineorreet  answer. 
Having  the  possible  ehoiees  listed  on  the  sereen  would  help  the  student  know  whieh  form 
of  an  answer  is  required. 

Aesthetic  and  Minimalist  Design 

The  fifth  hypothesis  looked  for  a  relationship  between  eourse  eompletion  rates 
and  aesthetie  and  minimalist  design  eharaeteristies.  There  were  three  eharaeteristies 
evaluated  for  this  study. 

Information  Essential  for  Decision-Making.  Courses  that  seored  low  on  this 
eharaoteristie  oontained  numerous  textboxes  in  the  review  questions  that  would  be 
rendered  with  stray  text  displayed  in  them.  This  text  was  often  an  answer  choioe  from  on 
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of  the  previous  questions  and  could  be  confusing  when  it  had  no  relation  to  the  question 
being  asked.  This  was  most  noticeable  in  the  Contract  Repair  Process  course.  Medium 
scores  were  given  to  courses  where  large  amounts  of  text  were  presented  on  a  single 
screen  that  covered  more  than  one  topic.  Limiting  the  information  on  a  page  to  one  topic 
may  give  the  student  more  time  to  internalize  the  information  before  having  to  think 
about  something  else. 

Brief  Menu  Titles.  There  should  not  be  more  information  in  a  menu  title  than  there 
is  on  the  rest  of  the  page.  None  of  the  courses  evaluated  appeared  to  have  a  problem  with 
the  length  of  menu  titles;  they  were  all  brief  and  succinct. 

Color  Used  With  Discretion.  Courses  that  received  low  scores  on  this 
characteristic  had  background  patterns  that  blended  with  the  text,  making  it  difficult  to 
read.  The  worst  example  of  this  was  the  AFRL  R&D  Case  File  Management  course 
which  had  a  medieval  castle  style  block  pattern  that  blended  with  every  fourth  line  of 
text;  so  much  so  that  the  text  was  almost  invisible.  Courses  receiving  a  medium  score  on 
this  characteristic  used  background  colors  that  were  bright  and  had  colored  text  that  did 
not  contrast  well  with  the  background  color.  After  spending  an  hour  or  so  looking  at  these 
colors,  my  eyes  began  to  ache.  Recommend  using  white  or  very  light,  pale  colors  with 
dark  fonts  to  help  give  some  contrast  to  the  text. 
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Help  and  Documentation 


The  sixth  hypothesis  looked  for  a  relationship  between  eourse  eompletion  rates 
and  help  and  doeumentation  design  faetors.  Four  eharaoteristies  were  evaluated  for  this 
study. 


Help  Function  Visible.  In  order  for  a  student  to  use  a  help  function,  there  must  be 
some  visible  way  of  accessing  it.  Low  scores  for  this  characteristic  were  given  to  courses 
that  had  no  visible  link  to  the  help  function  on  any  of  the  screens  within  the  course.  There 
were  seven  courses  that  received  a  score  of  one  on  this  characteristic.  Courses  that 
provided  a  help  link  on  at  least  the  menu  module  page  received  a  medium  score  on  this 
characteristic.  High  scores  were  given  to  courses  that  provided  a  link  on  every  page 
within  the  course. 

Information  Relevant.  For  a  help  system  to  be  useful,  it  should  provide  the  student 
with  help  in  accomplishing  whatever  task  they  are  having  problems  with.  This  help 
should  no  be  limited  to  how  to  use  the  VSH  system;  but  should  include  content  specific 
help  wherever  it  makes  sense.  Low  scores  on  this  characteristic  were  given  to  courses 
where  the  only  available  help  was  the  system  requirements  and  tutorial  on  how  to 
navigate  the  courses.  Medium  scores  were  given  to  courses  that  included  some  kind  of 
content  related  help  such  as  an  acronym  list,  or  a  list  of  references.  High  scores  were  give 
to  courses  that  provided  all  of  these  forms  of  help. 


47 


Easy  to  Access  and  Return  From  Help.  Once  a  student  accesses  the  help  system, 
they  should  not  have  to  navigate  all  the  way  baek  through  the  menu  system  to  return  to 
the  page  they  accessed  the  help  system  from.  Low  scores  on  this  characteristic  were 
given  to  courses  where  the  student  had  to  navigate  to  some  other  page  in  order  to  find  the 
help  system.  Medium  scores  were  given  when  the  student  could  access  the  help  system, 
but  the  navigation  functions  took  the  student  off  to  other  pages  in  the  system  rather  than 
back  to  the  page  they  were  on  when  they  invoked  the  help  system.  High  seores  were 
given  to  courses  that  displayed  the  help  in  a  separate  window  so  the  user  could  just  close 
it  and  the  window  they  were  on  is  still  showing. 

Errors  in  Matching  Exercises  Using  Connecting  Lines.  This  characteristic  only 
applied  to  the  Contract  Repair  Proeess  eourse.  This  course  had  an  exercise  that  drew 
lines  between  graphics  to  show  how  the  student  had  matched  them  up.  The  problem  was, 
the  lines  drew  very  slowly  and  caused  significant  delays  in  completing  the  exercise. 
There  were  many  other  examples  of  matehing  exereises  that  had  no  problems  with  slow 
refresh  rates.  I  would  reeommend  changing  this  exercise  to  one  of  those  methods.  By 
doing  so,  this  evaluation  eharacteristie  would  no  longer  be  needed  and  eould  be  removed 
from  future  evaluations  of  the  VSH  system. 
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Skills 


The  seventh  hypothesis  looked  for  a  relationship  between  eourse  completion  rates 
and  a  user’s  computer  skills.  There  were  four  skill  related  characteristics  evaluated  for 
this  study. 

Window  Operations  Easy  to  Learn  and  Use.  Generally,  the  window  operations  in 
the  VSH  were  easy  to  learn  and  use.  The  major  problem  with  using  the  VSH  system  was 
that  some  courses  would  redraw  the  homepage  screen  inside  the  course  frame  when 
returning  to  the  module  menu  screen.  After  three  or  four  iterations  through  the  modules, 
the  screen  would  be  full  of  main  menus  and  the  module  menus  would  no  longer  be 
visible.  After  some  trial  and  error,  I  found  that  the  screen  could  be  cleared  by  clicking  on 
the  “My  Classes”  icon  on  the  left  side  of  the  screen.  This  is  a  major  problem  that  can 
prevent  a  less  experienced  user  from  completing  the  courses  that  have  this  problem. 

Support  for  Novice  and  Expert  Users.  This  characteristic  deals  with  providing 
shortcuts  and  hidden  commands  that  will  allow  more  experienced  users  to  use  the  system 
faster.  There  was  nothing  observed  in  the  courses  that  appeared  to  implement  anything 
like  this.  Recommend  excluding  this  characteristic  from  future  evaluations  of  the  VSH 
system. 


Cursor  Positioned  in  Textbox.  A  course  scored  low  on  this  characteristic  if  a 
textbox  appeared  on  the  screen  without  positioning  the  cursor  in  it.  This  causes  the  user 
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to  waste  a  small  amount  of  time  having  to  position  the  cursor  before  they  can  begin 
typing.  This  distraction  hindered  easy  operation  of  the  VSH  system. 

Use  of  Chunking 

The  eighth  hypothesis  looked  for  a  relationship  between  course  completion  rates 
and  module  length.  Module  length  had  the  strongest  correlation  to  course  completion 
rates  of  all  heuristics  measured.  The  data  point  for  the  Introduction  to  Evolutionary 
Acquisition  course  was  removed  after  it  was  confirmed  as  an  outlier  from  all  other 
courses.  Removal  of  this  data  point  had  a  significant  effect  on  the  statistical  reliability  of 
module  length,  changing  it  from  -.17  to  -.51.  These  findings  parallel  Reynold’s  (2002) 
observations  of  module  length’s  effects  on  motivation  to  complete  an  e-learning  course. 
Determination  of  an  optimal  module  length  that  allows  students  to  complete  a  module 
before  becoming  distracted  could  have  a  positive  effect  on  completion  rates. 

Don  V  Lie  to  the  User 

This  item  was  not  statistically  significant.  This  could  mean  that  the  information 
that  these  links  are  meant  to  provide  is  either  irrelevant  or  unnecessary.  These  findings 
suggest  that  the  additional  effort  and  cost  involved  in  programming  many  hyperlinks  into 
the  courses  may  not  be  necessary.  Students  may  be  getting  all  of  the  information  they 
require  from  the  main  course  materials. 
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Summary  of  Findings 

Three  of  the  nine  hypotheses  were  supported.  Two  of  the  measures  for  eourse 
diffieulty,  readability  and  grade  level,  were  unreliable  and  should  not  be  used  in  future 
researeh.  The  original  HEC  used  for  the  evaluations  eontained  29  items.  The  revised 
HEC  now  has  22  evaluation  items  with  some  heuristies  eontaining  only  one  or  two  items. 
Euture  researeh  needs  to  be  eondueted  to  provide  additional  eharaeteristies  for  these 
heuristies. 

Implications 

Erom  a  theoretieal  standpoint,  this  study  has  shown  some  support  for  using 
heuristie  evaluation  to  identify  eourse  design  faetors  that  influenee  e-leaming  eourse 
eompletion  rates.  This  study  has  shown  that  heuristie  evaluation  is  a  low  eost,  effeetive, 
and  easy  to  use  tool  that  ean  identify  important  areas  where  eourse  designers  and 
administrators  ean  foeus  scaree  maintenanee  resourees  to  improve  the  usability  of  the 
eourses.  Module  length  was  one  area  that  appeared  to  have  signifieant  potential  to 
improve  eourse  usability.  Two  other  areas,  user  eontrol  and  freedom,  and  help  and 
doeumentation,  also  proved  to  be  areas  for  potential  improvements.  The  Air  Eoree  has 
indieated  a  desire  to  invest  in  training  opportunities  through  online  training  and  tools  like 
heuristie  evaluation  may  help  maximize  the  return  on  that  investment. 
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Limitations 


The  most  notable  limitation  of  this  study  was  the  application  of  the  heuristic 
evaluation  method  by  a  single  evaluator.  Nielsen  (1994)  states  while  a  single  evaluator 
can  perform  the  evaluation,  he/she  will  find  less  than  35  percent  of  the  usability  problems 
in  an  interface,  and  that  heuristic  evaluations  are  most  effective  when  conducted  by  three 
to  five  evaluators. 

Another  limitation  of  this  study  was  the  unvalidated  heuristic  evaluation 
instrument  used  to  rate  the  characteristics.  One  surprise  observation  was  the  statistically 
significant  inverse  relationship  between  course  completion  rates  and  H7.  A  positive 
relationship  was  expected  and  possible  explanations  for  this  are  choosing  the  wrong 
characteristics  to  measure  test  this  hypothesis  or  improperly  scoring  these  characteristics. 

A  third  limitation  is  the  Flesch  Reading  Ease  tool.  This  tool  only  measures  the 
surface  characteristics  of  the  text  such  as  sentence  length,  word  length,  and  syllables. 
These  characteristics  may  not  be  the  proper  items  to  test  on  web-based  e-learning 
courses.  Finding  a  better  tool  to  measure  the  comprehensibility  of  the  material  may 
provide  more  meaningful  results. 

A  final  limitation  of  the  study  was  the  Flesch-Kincaid  Grade  Fevel  tool.  Analysis 
of  the  data  showed  that  a  value  of  twelve  seemed  to  be  the  maximum  value  recorded.  It 
was  found  that  the  tool  in  Microsoft  Word  used  to  compute  this  value  only  reports  a 
maximum  value  of  twelve.  This  means  the  data  could  be  under-reported  for  this 
measurement  and  should  not  be  used  to  draw  any  conclusions. 
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Future  Research 


While  the  heuristie  evaluation  method  proved  to  be  useful  in  identifying  aspeets 
of  eourse  design  that  ean  improve  e-learning  eourse  usability,  further  researeh  using 
heuristies  to  evaluate  e-learning  eourse  design  is  needed  to  refine  and  extend  the 
eharaeteristies  used  to  evaluate  the  eourses.  This  study  has  provided  a  good  start  for  this 
line  of  researeh. 
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Appendix  A.  Heuristic  Evaluation  Checklist 


Visibility  of  System  Status 

The  system  should  always  keep  user  informed  about  what  is  going  on,  through 
appropriate  feedback  within  reasonable  time. _ 


# 

Review  Checklist 

Almost 

Almost 

Never 

Sometimes 

Always 

1.1 

Does  every  display  begin  with  a  title 
or  header  that  describes  screen 
contents? 

0 

0 

0 

0 

0 

1.2 

If  pop-up  windows  are  used  to  display 
messages,  do  they  allow  the  user  to 
see  the  field  in  error? 

0 

0 

0 

0 

0 

1.3 

Is  there  some  form  of  system  feedback 
for  every  operator  action? 

0 

0 

0 

0 

0 

1.4 

After  the  user  completes  a  group  of 
actions  (e.g.  a  module),  does  the 
feedback  indicate  that  the  next  group 
of  actions  can  be  started? 

0 

0 

0 

0 

0 

1.5 

If  an  object  is  to  be  moved,  is  there 
visual  feedback  on  where  to  put  the 
object? 

0 

0 

0 

0 

0 

1.6 

If  there  are  observable  delays  (greater 
than  fifteen  seconds)  in  the  system’s 
response  time,  is  the  user  kept 
informed  of  the  system's  progress? 

0 

0 

0 

0 

0 

1.7 

If  users  must  navigate  between 
multiple  screens,  does  the  system  use 
context  labels,  menu  maps,  and  place 
markers  as  navigational  aids? 

0 

0 

0 

0 

0 

NOTE:  Strikethrough  text  indicates  item  recommended  for  removal  from  HEC. 
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Match  Between  System  and  the  Real  World 

The  system  should  speak  the  user’s  language,  with  words,  phrases  and  eoneepts  familiar 
to  the  user,  rather  than  system-oriented  terms.  Follow  real-world  conventions,  making  information 
appear  in  a  natural  and  logical  order. _ 


# 

Review  Checklist 

Almost  Almost 

Never  Sometimes  Always 

2.1 

For  question  and  answer  interfaees, 
are  questions  stated  in  clear,  simple 
language? 

0  0  0  0  0 

NOTE:  Strikethrough  text  indieates  item  reeommended  for  removal  from  HEC. 
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User  Control  and  Freedom 


Users  should  be  free  to  seleet  and  sequenee  tasks  (when  appropriate),  rather  than  having 
the  system  do  this  for  them.  Users  often  ehoose  system  funetions  by  mistake  and  will 
need  a  elearly  marked  “emergency  exit”  to  leave  the  unwanted  state  without  having  to  go 
through  an  extended  dialogue.  Users  should  make  their  own  decisions  (with  clear 


information)  regarding  the  costs  of  exiting  current  work.  The  system  should  support  undo 
and  redo. 


# 

Review  Checklist 

Almost 

Never 

Sometimes 

Almost 

Always 

3.1 

In  courses  that  use  overlapping 
windows,  is  it  easy  for  users  to 
rearrange  windows  on  the  screen? 

0 

0 

0 

0 

0 

3.2 

In  courses  that  use  overlapping 
windows,  is  it  easy  for  users  to 
switch  between  windows? 

0 

0 

0 

0 

0 

3.3 

When  a  user's  task  is  complete,  does 
the  system  wait  for  a  signal  from  the 
user  before  processing? 

0 

0 

0 

0 

0 

^^9 

If  the  system  has  multiple  menu 
levels,  is  there  a  mechanism  that 
allows  users  to  go  back  to  previous 
menus? 

0 

0 

0 

0 

0 

3.5 

If  the  system  uses  a  question  and 

answer  interface,  can  users  go  back 
te — previous — questions — er — skip 
forward  to  later  questions? 

0 

0 

0 

0 

0 

3.6 

Are  vertical  and  horizontal  scrolling 
possible  in  each  window? 

0 

0 

0 

0 

0 

NOTE:  Strikethrough  text  indicates  item  recommended  for  removal  from  HEC. 


56 


Consistency  and  Standards 

Users  should  not  have  to  wonder  whether  different  words,  situations,  or  actions  mean  the 
same  thing.  Follow  platform  conventions. _ 


# 

Review  Checklist 

Almost 

Never 

Sometimes 

Almost 

Always 

4.1 

Have  formatting  standards  been 
followed  consistently  in  all  screens 
within  a  course? 

0 

0 

0 

0 

0 

4.2 

Has  a  heavy  use  of  all  uppercase 
letters  on  a  screen  been  avoided? 

0 

0 

0 

0 

0 

la 

Are  icons  labeled? 

0 

0 

0 

0 

0 

Are  attention-getting  techniques 
used  with  care? 

0 

0 

0 

0 

0 

1 

For  question  and  answer  interfaces, 
are  the  valid  inputs  for  a  question 
listed? 

0 

0 

0 

0 

0 

NOTE:  Strikethrough  text  indicates  item  recommended  for  removal  from  HEC. 
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Aesthetic  and  Minimalist  Design 


Dialogues  should  not  contain  information  which  is  irrelevant  or  rarely  needed.  Every 
extra  unit  of  information  in  a  dialogue  competes  with  the  relevant  units  of  information 
and  diminishes  their  relative  visibility. _ 


# 

Review  Checklist 

Almost 

Almost 

Never 

Sometimes 

Always 

5.1 

Is  only  (and  all)  information 
essential  to  decision  making 
displayed  on  the  screen? 

0 

0 

0 

0 

0 

5.2 

Are  menu  titles  brief,  yet  long 
enough  to  communicate? 

0 

0 

0 

0 

0 

5.3 

Has  color  been  used  with  discretion? 

0 

0 

0 

0 

0 

Help  and  Documentation 

Even  though  it  is  better  if  the  system  can  be  used  without  documentation,  it  may  be 
necessary  to  provide  help  and  documentation.  Any  such  information  should  be  easy  to 
search,  focused  on  the  user’s  task,  list  concrete  steps  to  be  carried  out,  and  not  be  too 
large. _ 


# 

Review  Checklist 

Almost 

Never 

Sometimes 

Almost 

Always 

6.1 

Is  the  help  function  visible;  for 
example,  a  key  labeled  KEEP  or  a 
special  menu? 

0 

0 

0 

0 

0 

6.2 

Is  the  information  relevant? 

0 

0 

0 

0 

0 

6.3 

Is  it  easy  to  access  and  return  from 
the  help  system? 

0 

0 

0 

0 

0 

6.4 

T7rsr*  w»ot-r»ninrT  ovor*r»iooo  fnof 

X  V/X  V'./V.V'X  V'XkJV'iJ  XXXXXX  V'V/XXXXV'V' t- 

ncif^rt  i -p  iicof 

0 

0 

0 

0 

0 

iXllo  TV  k/lkj  IX  Lllk/  Likjk/l 

on  or*i^  Imi^c 

iliu.iv^9  dii  tiiUij  Cli  t  k/UiiiitC/UUi  iiiit5 

fo/Hfoixm  m  n  fon  nnl  o  nm/-vnnt-  /-vP 

iCU.iU.VVii  iii  U  iCUaUiiUUiC  UiiiUU-iii  Ui 

time? 

NOTE:  Strikethrough  text  indicates  item  recommended  for  removal  from  HEC. 
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Skills 

The  system  should  support,  extend,  supplement,  or  enhance  the  user’s  skills,  background 


knowledge,  and  expertise  — not  replace  them. 


# 

Review  Checklist 

Almost 

Never 

Sometimes 

Almost 

Always 

7.1 

Are  window  operations  easy  to  learn 
and  use? 

0 

0 

0 

0 

0 

7.2 

If  the  system  supports  both  novice 

0 

0 

0 

0 

0 

of  detail  available. 

7.3 

When  the  user  enters  a  screen  or 
dialog  box,  is  the  cursor  already 
positioned  in  the  field  users  are  most 
likely  to  need? 

0 

0 

0 

0 

0 

NOTE:  Strikethrough  text  indicates  item  recommended  for  removal  from  HEC. 


59 


Flesch-Kincaid 
Grade  Level 


Flesch-Kincaid 
Grade  Level 

1 


1 
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34.3 
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12 
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21.9 

12 

Page  1 1 

192 

25.4 

12 

Page  12 

138 

24.3 

12 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  9 

Module  9 
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Page  1 
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Page  2 
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38.9 

11 

Page  3 
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41.7 
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200 

44.8 
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254 
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12 
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31 

12 
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46.2 

10.7 
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52.5 

9.7 
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146 

35.5 

11.8 

Page  1 1 

169 

42 

11.5 

Page  12 

239 

47.2 

10.1 

Page  13 

165 

36.8 

11.2 

Page  14 

233 

45.7 

10.4 

Page  15 

179 

28.5 

12 

Page  16 

111 

25.7 

12 

Page  17 

135 

4.2 

12 

Page  18 

142 

25.4 

12 

Page  19 

89 

13 

12 

Page  20 

204 

15.9 

12 

Page  21 

217 

15.8 

12 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  13 

Page  1 

58 

45.8 

11.4 

Page  2 

150 

36 

12 

Page  3 

113 

25.9 

12 

Page  4 

127 

45.3 

11.7 

Page  5 

120 

40.6 

12 

Page  6 

58 

27.3 

12 

Totals 

22424 

26.08 

11.79 

Avg.  Words/Module 

1725 

Useability  Items 

Review  Questions 

Mod  1  -  Told  to  choose  from  a  list,  but  must  type  the  item  into  a  text 
box. 

Review  Questions 

No  feedback.  Java  script  function  missing. 

Link  unavailable 

Mod  2,  Page  2  -  Contract  Repair  Guide,  para.  4.1 

Link  unavailable 

Mod  3,  Pg  1  -  Contract  Repair  Process  (CRP) 

CRP  Exercise 

Difficult  to  locate  where  to  drop  images.  Would  pop  around  to  wrong 
places. 

CRP  Diagram 

A  lot  of  links  to  diagram,  but  diagram  doesn't  show  where  information 
fits  into  process 
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Confusing  Exercise 

Mod  4  Applet  TitleMgr  exercise  very  confusing.  No  feedback  when 
linking  items. 

Progress  indicator 

No  indication  of  progress  in  long  module  4 

Link  location 

Sent  to  class  listing  page  after  summary  in  module  5  instead  of  course 
page. 

Module  6.5  review 

Review  Quiz  does  not  automatically  let  me  change  an  incorrect 

quiz 

answer 

Module  6.5  Summary 

System  D200A  was  discussed  and  tested  on  but  not  mention  in  the 
summary 

Module  7  Objectives 

Qbjectives  page  shows  objectives  for  Mods  4, 5, 6, and  7  but  other 
objective  pages  only  show  that  modules  objectives 

Module  7  Review 

Asks  user  to  choose  from  a  list  but  forces  user  to  type  in  choice.  Why 

Question 

not  dropdown  list? 

Module  7.2 

Reference  Lightning  Bolt  10  link  broken. 

System  Down 

System  went  down  during  evaluation  -  1208PM 

Module  7.3 

Market  Research  link  resulted  in  requested  URL  not  found  error. 

Mod  7  Quiz 

User  typing  in  answers  can  result  in  wrong  answer  if  spelling  is 

Questions 

incorrect  even  if  attempted  answer  was  correct 

Mod  7.3  Developing 
the  contract 

Second  sentence-'customary  commercial  practices"  is  will  be  a 

Mod  8.1  Matching 

Confusing.  Lines  draw  so  slowly  that  student  doesn't  know  what  they 

quiz 

are  until  4th  try 

Mod  8.2  Review 

Error  checking  logic  appears  to  be  in  applet.  Applet  takes  too  long  or 

qstns 

never  downloads 

Mod  8.3  Contract 
Types 

Acquisition  Reform  Initiatives  link  broken 

Mod  8.3  Contract 
Types 

Cost  Benefit  Analysis  link  broken 

Mod  9.1 

Process  Action  team  link  broken.  Appears  to  be  a  glossary  page  that 
is  unavailable. 

Mod  9.1 

Acq  Policy  Memo  99-1  link  broken 

Mod  9.5 

SAF/AQ  policy  98A-001  link  broken 

Mod  9.5 

CID  glossary  lookup  link  broken 

Mod  9.7 

HQ  AFMC  SQQ/SQW  link  broken 

Mod  10 

Contract  Repair  link  requires  credentials  that  are  not  available 

Mod  10.2 

data  management  link  broken 

Mod  10.3 

Data  Item  Description  link  broken 

Mod  10.3 

Data  Accession  List  link  broken 

Mod  11.2.1 

Small  Business  Administration  link  broken 

Mod  11.2.1 

Contract  Repair  Team  link  broken 

Mod  11 

Review  question  answer  boxes  contain  extraneous  characters 

Mod  11.2.3 

Early  Contract  Administration  Services  link  broken 

Mod  11.2.3 

Contract  Administration  Qffices  link  broken 

Mod  11.2.3 

"Qne  Book"  link  requires  special  access  login 

Mod  11.2.4 

DoD  CAS  Directory  link  requires  special  access  login 

Mod  12.1 

TRACKER  link  broken 

Mod  12.3 

WorldWideExpress  link  requires  special  access  login 
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Mod  12.3 

Foreign  Clearance  Guide  requires  special  credentials 

Mod  13.1 

Contractor  Furnished  Material  link  broken 

Mod  13.1 

Government  Furnished  Material  link  broken 

Mod  13.1 

Contract  Line  Item  (CLIN)  link  broken 

Total  Links 

150 

Broken  Links 

34 

Broken  Links  % 

23% 
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19.8 

12 

Page  8 

162 

19.7 

12 

Page  9 

371 

24 

12 

Page  10 

307 

31.6 

12 

Page  1 1 

316 

33.9 

12 

Page  12 

231 

25.5 

12 

Page  13 

226 

46.7 

10.2 

Page  14 

162 

18 

12 
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Totals 

28015 

36.90 

11.30 

Avg.  Words/Module 

2547 

Mod  1.1 

Business  Management  Plan  link  broken 

Mod  1.1 

Strategic  Plan  for  FY2002-2009  link  broken 

Mod  1 .4 

nine  mission  essential  tasks  link  broken 

Mod  8.2 

Not  authorized  to  view  content  behind  Air  Force  Strategic  Vision  link 

Mod  8.2 

Not  authorized  to  view  content  behind  AF/XPP  webpage  link 

Mod  8.2 

PPBS  link  opens  new  window  without  informing  user  or  including  close 
button... User  must  figure  out  where  they  are 

Mod  8.2 

Not  authorized  to  view  content  behind  Annual  Planning  and 
Programming  Guidance  link 

Mod  8.2 

AFMC  Strategic  Plan  link  broken 

Mod  8.2 

Mission  Essential  Task  List  link  broken 

Mod  8.2 

AFMC  corporate  management  structure  mission  area  links  (8)  broken 

Mod  8.2 

AFMC  Strategic  Planning  link  broken 

Mod  8.3 

AFMC  Mission  Essential  Tasks,  Sep  98  link  broken 

Mod  10.1 

HQ  AFMC/XP  website  link  broken 

Mod  10.2 

HQ  AFMC/FM  website  link  broken 

Total  Links 

117 

Broken  Links 

14 

Broken  Links  % 

12% 
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Course  Name: 

Introduction  to  Risk  Management 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  1 

Page  1 

1367 

51.8 

10.7 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  2 

Page  1 

238 

24.2 

12 

Page  2 

330 

26.6 

12 

Page  3 

504 

45.5 

11 

Page  4 

393 

38.8 

11 

Page  5 

379 

36.6 

11.4 

Page  6 

299 

36.7 

12 

Page  7 

132 

42.1 

11.8 

Page  8 

251 

41.6 

11 

158 

46.7 

10.6 

412 

20.3 

12 

Page  1 1 

130 

0 

12 

Page  12 

274 

20.6 

12 

Page  13 

54 

30.5 

12 

Page  14 

297 

14.6 

12 

Page  15 

138 

9.8 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  3 

Page  1 

74 

22.8 

12 

Page  2 

210 

33.1 

12 

Page  3 

260 

16.7 

12 

Page  4 

216 

28.6 

11.9 

Page  5 

194 

30.6 

12 

Page  6 

282 

18.9 

12 

Page  7 

252 

17 

12 

Page  8 

228 

19.1 

12 

Page  9 

145 

28.9 

12 

Page  10 

277 

18.4 

12 

Page  1 1 

89 

53.7 

8.9 

Page  12 

171 

15.3 

12 

Page  13 

209 

32.6 

12 

Page  14 

456 

18.8 

12 

Page  15 

130 

0 

12 
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Page  16 

232 

29.1 

12 

Page  17 

171 

44.3 

11.3 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  4 

Page  1 

308 

32.3 

12 

Page  2 

254 

21.6 

12 

Page  3 

211 

13.4 

12 

Page  4 

144 

35.2 

11.5 

Page  5 

34 

0 

0 

Page  6 

152 

44.9 

10.4 

Page  7 

152 

41.1 

12 

Page  8 

118 

15.3 

12 

Page  9 

118 

2.6 

12 

Page  10 

201 

36.5 

12 

Page  1 1 

167 

15.1 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  5 

Page  1 

87 

40.8 

12 

Page  2 

229 

40.9 

10.9 

Page  3 

230 

51.3 

8.9 

Page  4 

286 

45.4 

10.9 

Page  5 

150 

26.9 

12 

Page  6 

116 

45.5 

10.7 

Page  7 

229 

41.4 

12 

Page  8 

272 

27.9 

12 

Page  9 

319 

46.2 

10.3 

Page  10 

186 

30.5 

12 

Page  1 1 

115 

18.9 

12 

Page  12 

180 

38.5 

10.5 

Page  13 

226 

54.2 

9.2 

Page  14 

137 

54 

8.7 

Page  15 

128 

44.9 

10.4 

Page  16 

354 

45.9 

11.2 

Page  17 

165 

25.7 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  6 

Page  1 

232 

30.6 

11.7 

Page  2 

216 

38 

11.4 

Page  3 

272 

62.4 

8.2 

73 


Page  4 

179 

32.6 

11.6 

Page  5 

167 

37.9 

11.1 

Page  6 

262 

39.3 

11.4 

Page  7 

242 

45.7 

11.1 

Page  8 

365 

34.2 

12 

Page  9 

60 

33.4 

11.4 

Page  10 

173 

25.9 

12 

Page  1 1 

250 

43 

11.1 

Page  12 

168 

25.5 

12 

Page  13 

199 

31.9 

11.7 

Page  14 

164 

34.5 

12 

Page  15 

117 

50.4 

10.2 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  7 

Page  1 

194 

34.9 

12 

Page  2 

158 

41.6 

10.8 

Page  3 

438 

31.6 

12 

Page  4 

265 

42.6 

10.5 

Page  5 

181 

25.4 

12 

Page  6 

288 

29.5 

12 

Page  7 

283 

41.8 

11.2 

Page  8 

120 

46 

9.9 

Page  9 

171 

40.6 

10.1 

Page  10 

210 

37.4 

11.8 

Page  1 1 

183 

42.1 

11.6 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  8 

Page  1 

154 

24.3 

12 

Page  2 

83 

30.3 

11.5 

Page  3 

335 

48.4 

9.8 

Page  4 

441 

45.4 

9.5 

Page  5 

74 

46.9 

10.1 

Page  6 

327 

47.8 

9.7 

Page  7 

163 

46.5 

10.8 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  9 

Page  1 

158 

39.9 

11.3 

74 


Page  2 

191 

23.2 

12 

101 

4.5 

12 

Page  4 

241 

34.4 

12 

Page  5 

146 

47.1 

10.3 

Page  6 

230 

29.3 

12 

Page  7 

371 

31.7 

12 

Page  8 

411 

8.1 

12 

Page  9 

234 

34.8 

12 

Page  10 

545 

30.9 

12 

Page  1 1 

245 

24.7 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  10 

Page  1 

344 

35.2 

12 

Page  2 

429 

46 

9.7 

Page  3 

142 

54 

7.7 

Page  4 

146 

36.9 

10.9 

Page  5 

142 

31.5 

12 

Page  6 

31 

30 

12 

Page  7 

185 

51.2 

9.1 

Page  8 

123 

54.7 

8.4 

Page  9 

188 

48.9 

9.1 

Page  10 

141 

56.1 

7.8 

Page  1 1 

255 

53.9 

9.7 

Page  12 

149 

49.8 

9.9 

Page  13 

93 

63.6 

7.7 

Page  14 

171 

24.4 

12 

Page  15 

138 

54 

7.2 

Page  16 

183 

37.5 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  11 

Page  1 

628 

22.7 

12 

Page  2 

250 

33.1 

12 

Page  3 

305 

30.8 

12 

Page  4 

304 

43.8 

10.7 

Page  5 

202 

17.1 

Page  6 

367 

33.3 

Page  7 

84 

29.5 

Totals 

29227 

34.11 

Avg.  Words/Module 

2657 
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Usability  Items 

Module  2.7 

Mandatory  Procedures  for  major...  link  broken 

Module  2.7 

Acquisition  Risk  Management  Guide  link  broken 

Module  2.11 

DOD  Risk  Management  Homepage  link  broken 

Module  3.9 

DTSE&E  Risk  Management  link  broken 

Module  3.11 

DODD  5000.1  "Defense  Acquisition"  link  broken 

Module  3.11 

DODD  5000. 2R  "mandatory..."  link  broken 

Module  4.3 

DSMC  Risk  Management  Guide  link  broken 

Module  4.3 

AFMC  Acquisition  Risk  Management  Pamphlet  link  broken 

Module  4.7 

DODD  5000.1  "Defense  Acquisition"  link  broken 

Module  4.7 

DODD  5000.2R  link  broken 

Module  5.8 

Pareto  Chart  link  broken 

Module  5.10 

"AFMC  Pamphlet  63-101..."  link  broken 

Module  6.8 

"AFMC  Pamphlet  63-101..."  link  broken 

Module  7.6 

"AFMC  Pamphlet  63-101..."  link  broken 

Module  7.6 

Risk  Management  Guide  link  broken 

Module  7.6 

NAVSD-p-3606...  link  broken 

Module  8.6 

DODD  5000.1  "Defense  Acquisition"  link  broken 

Module  8.6 

DODD  5000.2R  link  broken 

Module  8.6 

DSMC  Risk  Management  Guide  link  broken 

Module  8.6 

AFMC  Pamphlet  63-100  link  broken 

Module  9.4 

System  Maturity  Matrix  link  broken 

Module  9.4 

OSD  TPM  Webpage  link  broken 

Module  9.9 

AFMC  Pamphlet  63-100  link  broken 

Module  10.7 

AFMC  Pamphlet  63-100  link  broken 

Module  10.7 

DCMC  Earned  Value  Management  Systems  Center  link  broken 

Module  11.5 

DODD  5000.1  "Defense  Acquisition"  link  broken 

Module  11.5 

DODD  5000.2R  link  broken 

Module  11.5 

DSMC  Risk  Management  Guide  link  broken 

Module  11.5 

AFMC  Pamphlet  63-100  link  broken 

Total  Links 

188 

Broken  Links 

29 

Broken  Links  % 

0.15 
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Course  Name: 

Maintenance  Planning  Course 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  1 

Page  1 

100 

39.7 

11.6 

Page  2 

132 

55.2 

9.7 

Page  3 

96 

45.2 

10.2 

Page  4 

121 

46.7 

10.4 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  2 

Page  1 

147 

15.5 

12 

Page  2 

112 

22.1 

12 

Page  3 

89 

31.2 

12 

Page  4 

92 

17.3 

12 

Page  5 

152 

32.6 

12 

Page  6 

181 

12.9 

12 

Page  7 

43 

36.7 

11.4 

Page  8 

101 

38.9 

12 

Page  9 

56 

40.2 

11 

Page  10 

182 

17.9 

12 

Page  1 1 

86 

15.4 

12 

Page  12 

55 

27.4 

11.9 

Page  13 

115 

37 

12 

Page  14 

210 

24.5 

12 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  3 

Page  1 

101 

59.6 

8.4 

Page  2 

145 

24.2 

11.8 

Page  3 

198 

9.4 

12 

Page  4 

94 

11.3 

12 

Page  5 

135 

4.2 

12 

Page  6 

135 

11.7 

12 

Page  7 

208 

40.5 

11.5 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  4 

Page  1 

126 

16.3 

12 

Page  2 

161 

0 

12 

Page  3 

108 

61.2 

8.6 
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Page  4 

144 

19.2 

12 

Page  5 

309 

17.1 

12 

Page  6 

191 

43.8 

11.6 

Page  7 

168 

8.2 

12 

Page  8 

191 

36.1 

12 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  5 

Page  1 

248 

30.2 

12 

Page  2 

396 

33.5 

12 

Page  3 

199 

35.1 

12 

Page  4 

85 

35.4 

12 

Page  5 

131 

29.3 

12 

Page  6 

161 

51 

9.3 

Page  7 

147 

44.4 

11.2 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  6 

Page  1 

213 

50.7 

10.2 

Page  2 

377 

68.4 

7.1 

185 

43.2 

10.8 

Page  4 

191 

27.8 

12 

Page  5 

128 

19.7 

12 

Page  6 

62 

38.8 

10.6 

Page  7 

284 

28.8 

12 

Page  8 

158 

36.4 

12 

Total 

7449 

31.08 

11.40 

Avg.  Words/Module 

1242 

Usability  Items 

Module  2.1 

DOD  Directive  5000.1  link  redirected 

Module  2.1 

DOD  Instruction  5000.2  link  redirected 

Module  2.1 

DOD  Regulation  5000. 2R  link  redirected 

Module  2.2 

Flexible  Sustainment  Guide  link  broken 

Module  2.2 

Reliability  based  Logistics  link  broken 

Module  2.2 

Trigger  based  Asset  Management  link  broken 

Module  3.1 

Maintenance  Planning  Guide,  Section  2.1.1  link  broken 

Module  3.1 

Maintenance  Planning  Guide,  Section  2.1.2  link  broken 

Module  3.1 

Maintenance  Planning  Guide,  Section  2.1.3  link  broken 

Module  3.1 

Maintenance  Planning  Guide,  Section  2.1.4  link  broken 

Module  3.1 

Maintenance  Planning  Guide,  Section  2.1.5  link  broken 

Module  3.2 

Maintenance  Planning  Guide,  Section  2.2.1  link  broken 

Module  3.2 

Maintenance  Planning  Guide,  Section  2.2.2  link  broken 

Module  3.2 

Maintenance  Planning  Guide,  Section  2.2.3  link  broken 
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Module  3.2 

Maintenance  Planning  Guide,  Section  2.2.4  link  broken 

Module  3.2 

Maintenance  Planning  Guide,  Section  2.2.5  link  broken 

Module  3.2 

Maintenance  Planning  Guide,  Section  2.2.6  link  broken 

Module  3.3 

Maintenance  Planning  Guide,  Section  2.3.1  link  broken 

Module  3.3 

Maintenance  Planning  Guide,  Section  2.3.2  link  broken 

Module  3.4 

The  Logistics  Elements  7.2.2  Maintenance  Planning  link  broken 

Module  3.4 

The  Logistics  Elements  7.2.3  Manpower  and  Personnel  link  broken 

Module  3.4 

The  Logistics  Elements  7.2.4  Supply  Support  link  broken 

Module  3.4 

The  Logistics  Elements  7.2.5  Support  Equipment  link  broken 

Module  3.4 

The  Logistics  Elements  7.2.6  Technical  Data  link  broken 

Module  3.4 

The  Logistics  Elements  7.2.7  Training  and  Support  link  broken 

Module  3.4 

The  Logistics  Elements  7.2.8  Facilities  link  broken 

Module  3.4 

The  Logistics  Elements  7.2.9  Packaging,  Handling,  and  Storage  link 
broken 

Module  3.4 

The  Logistics  Elements  7.2.10  Computer  Resources  Support  link 
broken 

Module  3.4 

The  Logistics  Elements  7.2.11  Design  Interface  link  broken 

Module  4.1 

Maintenance  Planning  Guide  2.2.4  link  broken 

Module  5.2 

SORAP  Guide  link  broken 

Module  6.2 

Under  Title... Part  1  -  The  Schedule  link  broken 

Module  6.2 

Under  Title... Part  II  -  Contract  Clauses  link  broken 

Module  6.2 

Under  Title... Part  III  -  List  of  Documents,  Exhibits,  and  other 
attachments  link  broken 

Module  6.2 

Under  Title... Part  IV  -  Representations  and  Instructions  link  broken 

Module  6.2 

Under  Examples  of  Logistics  Inputs...  Part  1  -  The  Schedule  link 
broken 

Module  6.2 

Under  Examples  of  Logistics  Inputs...  Part  II  -  Contract  Clauses  link 
broken 

Module  6.2 

Under  Examples  of  Logistics  Inputs...  Part  III  -  List  of  Documents, 
Exhibits,  and  other  attachments  link  broken 

Module  6.2 

Under  Examples  of  Logistics  Inputs...  Part  IV  -  Representations  and 
Instructions  link  broken 

Module  6.2 

Under  Section...  C:  Description/Specifications/Work  link  broken 

Module  6.2 

Under  Section...  J:  List  of  Attachments  link  broken 

Module  6.2 

Under  Section...  L:  Instruction,  Conditions,  and  Notes  to  offerors  link 
broken 

Module  6.2 

Under  Section...  M:  Evaluation  Factors  for  Award  link  broken 

Module  6.4 

Section  8.4.11  of  Mil-HDBK-502  link  broken 

Module  6.4 

Maintenance  Planning  Guide:  Appendix  A,  Section  5.2.1  link  broken 

Module  6.4 

Maintenance  Planning  Guide:  Appendix  A,  Section  5.2.2  link  broken 

Module  6.4 

Maintenance  Planning  Guide:  Appendix  A,  Section  5.2.3  link  broken 

Module  6.4 

Maintenance  Planning  Guide:  Appendix  A,  Section  5.2.4  link  broken 

Module  6.4 

Maintenance  Planning  Guide:  Appendix  A,  Section  5.2.5  link  broken 

Module  6.4 

Maintenance  Planning  Guide:  Appendix  A,  Section  5.2.6  link  broken 

Total  Links 

149 

Broken  Links 

50 

Broken  Links  % 

34% 
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Course  Name: 

Pricing  Analysis  Methods 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module:  Course 

Overview 

Page  1 

112 

51.4 

8.6 

Page  2 

85 

28.7 

12 

Page  3 

172 

34.5 

11.8 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module:  Price  Analysis 
Methods  in  FAR 

Page  1 

52 

20.9 

12 

Page  2 

53 

27.4 

11.9 

Page  3 

155 

42 

10.2 

154 

45.8 

11 

Page  5 

50 

64.7 

10.4 

Page  6 

82 

51.3 

8.1 

Page  7 

178 

38.6 

11.6 

Page  8 

61 

30.8 

12 

Page  9 

177 

35.3 

11.4 

Page  10 

155 

27.6 

12 

Page  1 1 

185 

40.7 

11.2 

Page  12 

38.4 

12 

Page  13 

148 

38.6 

11.5 

Page  14 

51.4 

Page  15 

167 

43.5 

10.9 

Page  16 

45.8 

Page  17 

75 

59.6 

8.4 

56 

51.9 

8.7 

48 

67.9 

9.7 

108 

20.1 

12 

167 

49 

9.6 

Page  22 

113 

37.7 

12 

139 

53.3 

9.2 

Page  24 

162 

41.5 

11.4 

Page  25 

84 

20.8 

12 

Page  26 

99 

22.5 

12 

Page  27 

51.7 

9.4 

Page  28 

25.7 

12 

Page  29 

49 

66.3 

8.4 

Page  30 

64 

28.6 

12 

Page  31 

169 

41.4 

12 
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Page  32 

84 

40 

Page  33 

221 

51.4 

10.6 

Page  34 

110 

33.9 

12 

Page  35 

176 

28.9 

12 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module:  Value  Analysis 

Page  1 

58 

30.9 

12 

Page  2 

130 

30.1 

12 

Page  3 

63 

32.4 

12 

Page  4 

125 

47.9 

11 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module:  Performance 
Based  Payments 

Page  1 

103 

4.1 

9.7 

Page  2 

157 

24.4 

12 

Page  3 

121 

46.1 

10 

Page  4 

182 

43.4 

10.6 

Page  5 

97 

29.6 

12 

Page  6 

279 

37.8 

12 

Page  7 

166 

54.9 

9.7 

Page  8 

193 

51.5 

10.3 

Page  9 

123 

46 

10.8 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module:  Negotiations 

Page  1 

59 

17.4 

12 

Page  2 

73 

0 

12 

Page  3 

103 

42.5 

11.3 

Page  4 

110 

38.3 

11.5 

Page  5 

94 

45.6 

10.2 

Page  6 

152 

45.8 

11.9 

Page  7 

108 

30.9 

12 

Page  8 

51 

37.3 

9.8 

Total 

7145 

38.75 

10.97 

Avg.  Words/Module 

1429 

Usability  Items 

Course  Overview 

http://www.dodig.osd.mil/audit/reports/fy01/01-129pt1.pdf  link 
broken 

Course  Overview 

http://www.dodig.osd.mil/audit/reports/fy01/01-129pt2.pdf  link 
broken 
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Price  Analysis  Methods 
in  FAR 

Market  Research  Section  https://www.afmc  mil.wpafb.af.mil/HQ- 
AFMC/PK/PKP/PKPC/MR-CAQC.htm  link  broken 

Price  Analysis  Methods 
in  FAR 

Market  Research  Section 

https://afkm.wpafb.af.mil/ASPs/ACQ/PST/PSTHome.htm  link 
broken 

Price  Analysis  Methods 
in  FAR 

Market  Research  Section 

https://www.conconnect.wpafb.af.mil/welcome/welcome.asp  link 
broken 

Negotiations  Module 

https://www.afmc_mil.wpafb.af.mil/HQ-AFMC/PK/PKP/guide.htm 
link  broken 

Total  Links 

38 

Broken  Links 

6 

Broken  Links  % 

0.16 
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Course  Name: 

Introduction  to  CM 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  1 

Paqe  1 

128 

42.5 

11.3 

Page  2 

27 

42.7 

10.6 

Page  3 

74 

62.9 

7.4 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  2 

Page  1 

197 

22.5 

12 

Page  2 

126 

41 

11.2 

Page  3 

22 

72.6 

5.8 

Page  4.1  CP&M 

64 

18.9 

12 

Page  4.2  SA 

87 

0 

12 

Page  4.3  ID 

204 

1.2 

12 

Page  4.4  Ch.Cntr 

189 

17.7 

12 

Page  4.5  Ch.  Aud 

246 

10.4 

12 

Page  5  Benefits? 

41 

0 

12 

Page  6  Develop? 

116 

45.7 

11.6 

Page  7 

35 

80.3 

6.4 

Page  8  Summary 

100 

43.5 

10.2 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  3 

Page  1 

133 

22.9 

12 

Page  2 

320 

40.1 

11.7 

Page  2.1  Example 

95 

55.3 

11.4 

Page  3 

22 

57.2 

8 

Page  3.1  Writ  Procs 

250 

48.4 

12 

Page  3.2  CM  Value 

418 

45.6 

10.7 

Page  3.3  Consist 

167 

15.8 

12 

Page  3.4  CCB 

244 

23.7 

12 

Page  3.5  Context 

133 

49.3 

11.8 

Page  3.6  Mng  Act 

193 

31.4 

12 

Page  3.7  CM  Trng 

152 

26.3 

12 

Page  3.8  Cont  Eval 

146 

24.1 

12 

Page  4  Principles  ? 

63 

28.4 

12 

Page  5  Summary 

93 

54.5 

8.8 

84 


Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  4 

Page  1  Cm  Overview 

90 

21.5 

12 

2 

129 

0 

12 

3  Selecting  CIs 

159 

14.4 

12 

4 

176 

23.6 

12 

5  CIs 

184 

3.9 

12 

6 

249 

46.4 

10.8 

7 

79 

59.1 

7 

8  Baselines 

200 

37.3 

12 

8.1  Func  BsIn 

38 

27.8 

12 

8.2  A&P  BsIn 

149 

13.1 

12 

9  Specs 

251 

8.9 

12 

10  Acq  BSpecs 

234 

14.9 

12 

10.1  MilStd961 

374 

15 

12 

1 1  Sec3  Rqmnt 

101 

10.9 

12 

12  Sec4Verific 

188 

41.2 

11 

12.1  Qualification 

81 

2.9 

12 

12.2  First  Article 

84 

25.6 

12 

12.3  Acceptance 

58 

22.3 

12 

13  Methods 

100 

23.2 

12 

14  Int.  Defs 

251 

20.9 

12 

14.1  Sys  Perf  Specs 

78 

7.7 

12 

14.2  Item  Perf  Specs 

66 

8.2 

12 

14.3  Item  Det  Specs 

36 

0 

12 

14.4  Assymbly  Draw 

18 

21.3 

12 

14.5  Install  Draw 

37 

0 

12 

14.6  ICD 

57 

0 

12 

14.7  IRS 

33 

24.6 

12 

14.8  IDD 

36 

0 

12 

14.9  Src  Con  Draw 

50 

21.7 

12 

14.10  SPS 

35 

0.2 

12 

14.11  SoftDetSpec 

42 

2.6 

12 

14.12  SRS 

65 

2 

12 

15  Summary 

122 

38.8 

10.6 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  5 

Page  1 

243 

25.5 

12 

2 

332 

21.8 

12 

2.1  ECPs 

204 

19.7 

12 

3 

188 

25.9 

12 

3.1  RFDs 

169 

18.8 

12 

4  CCBs 

238 

30.2 

12 

85 


4.1  Mil  HNDBK61a 

238 

27.7 

12 

5  CCB  Chair 

221 

22.3 

12 

6.1  Pre  ECP 

101 

21.9 

12 

6.2  Write  ECP 

348 

22.7 

12 

6.3  Classes  of  ECPs 

221 

16.1 

12 

6.4  Types  of  ECPs 

201 

27.6 

12 

6.5  ECP  Priorities 

198 

26.1 

12 

6.6  ECP  Effectivity 

122 

20.5 

12 

6.7  CCB  Approves 

299 

33.9 

12 

6.8  Upd  Base  Docs 

104 

13.1 

12 

6.9  Upd  TO 

103 

12.2 

12 

6.10  Write  TCTOs 

53 

28.1 

12 

6.1 1  Upd  Provis  Data 

57 

38.1 

12 

6.12  Mod  Product 

14 

11.3 

12 

6.13  Mod  Spares 

14 

18.4 

12 

6.14  Other  Activities 

43 

34.6 

11.7 

7.1  Submit/App  ? 

91 

38 

11.7 

8.1  Write  RFD 

131 

41.1 

11.8 

8.2  Classes  of  Devs 

193 

9.5 

12 

8.3  Dev  Considerations 

220 

15.6 

12 

8.4  CCB  Approves 

340 

31.5 

12 

8.5  RFD 

Implementation 

144 

33.8 

12 

9  Summary 

106 

58.1 

8 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  6 

Page  1  CM  Roles 

141 

43.7 

12 

2  Roles  Cont 

190 

19.7 

12 

2.1  Req  Analysis 

29 

35.9 

10.7 

2.2  Func  Anal/Alloc 

64 

22.9 

12 

2.3  Synthesis 

27 

0 

12 

2.4  Sys  Analysis/Cont 

67 

19 

12 

3  Func  Reviews 

229 

32.2 

12 

3.1  ASR 

47 

11.8 

12 

3.2  SRR 

34 

8.3 

12 

3.3  SFR 

52 

12.5 

12 

3.4  PDR 

61 

6.4 

12 

3.5  SSR 

49 

28 

12 

3.6  CDR 

67 

16.3 

12 

3.7  TRR 

89 

22.8 

12 

3.8  SVR 

85 

27.5 

12 

3.9  FCA 

138 

11.6 

12 

3.10  PCA 

187 

18 

12 

4 

245 

24 

12 

86 


5  Summary 

113 

49.7 

9.6 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  7 

Page  1 

102 

17.5 

12 

1.1  Status  Acct. 

104 

21.3 

12 

2  CSA  Info  Media 

163 

34.5 

12 

3  CSA  Info  Media  Cent 

143 

25.7 

12 

4  CSA  Info  Media  Cent 

163 

24.8 

12 

5  CSA  Info  Media  Cent 

65 

36.7 

12 

5.1  CSA  Question 

35 

1.6 

12 

6  Data  Categories 

102 

40 

10.3 

7  Types  of  CSA  Data 

120 

36.3 

12 

7.1  PPO  Info 

208 

26 

12 

7.2  Devel  Info 

218 

12.8 

12 

7.3  Tech  Baseline  Docs 

118 

14.9 

12 

7.4  Chg  Cont  Docs 

156 

7.4 

12 

7.5  Rev,  Comment 

200 

27.3 

12 

7.6  Audit  Results 

135 

27.1 

12 

7.7  Oper  Assets 

177 

18.1 

12 

8  Elec  Data  Security 

272 

15.3 

12 

9.1  EDS  ? 

92 

40.1 

12 

10  Lifecycle  Support 

163 

59 

9.3 

11  Summary 

125 

47.2 

9.9 

Totals 

17646 

25.18 

11.58 

Avg.  Words/Module 

2521 

Total  Links 

100 

Broken  Links 

0 

Broken  Links  % 

0% 
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Course  Name: 

Operational  Safety,  Suitability,  and  Effectiveness 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  1 

Page  1  Instructions 

427 

45.2 

10.9 

1.1  Syllabus 

524 

36.3 

12 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  2 

Page  1  Events 

242 

14.9 

12 

2  Policy 

260 

29.3 

12 

3  Policy  Cont 

131 

1.9 

12 

4  Certifications 

263 

7.6 

12 

5  AF  Policy  Dir 

119 

34.5 

12 

6  Policy  Doc  Cont 

182 

16.3 

12 

7  Policy  Doc  Cont 

140 

28.1 

12 

8  Policy  Doc  Cont 

130 

25 

12 

9  OSS&E  Airworthiness 

204 

17.9 

12 

10  OSS&E  Toolkit 

141 

40.5 

12 

11  Summary 

83 

49 

11.3 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  3 

Page  1  HHQ  Roles 

278 

7.1 

12 

2  HHQ  Roles  Cont 

369 

14.2 

12 

3  HHQ  Roles  Cont 

153 

22.4 

12 

4  Single  Manager 

237 

16.2 

12 

5  Chief  Engineer 

247 

5.6 

12 

6  SLAs 

149 

26.7 

12 

7  SLAs  Cont 

175 

23.7 

12 

8  Summary 

117 

50.4 

10.1 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  4 

Page  1-OSS&E  Baseline 

248 

29.7 

12 

2  Definitions 

304 

0 

12 

3  Definitions  Cont 

246 

24.4 

12 

4  Product  Baseline 

198 

23.6 

12 

5  Product  Guides/inst 

340 

16.9 

12 

6  Baselines  to  Mng  systems 

134 

31.5 

12 

88 


7  Baselines  to  Mng  systems 

82 

2.4 

12 

8  Summary 

74 

25.2 

12 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  5 

Page  1  Disc  Engr  Process 

345 

0 

12 

2  Disc  Engr  Process  Cont 

152 

11.6 

12 

3  CM 

303 

25.1 

12 

4  Audits 

293 

14.3 

12 

5T&E 

203 

0 

12 

6-Other  Mandatory  Elements 

268 

0 

12 

7  -Systems  and  End  Items 

197 

33.6 

12 

8  -Systems  and  End  Items 

Cont 

123 

16.1 

12 

9  Metrics 

122 

54.7 

11.3 

10  Metrics  Cont. 

105 

0 

0 

1 1  Metrics  Cont. 

347 

36.6 

12 

12  Summary 

142 

28.6 

12 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  6 

Page  1  Enterprise  Overview 

320 

19.5 

12 

2  Enterprise  Overview  Cont 

304 

11.6 

12 

3  Enterprise  Overview  Cont 

289 

26 

12 

4  Enterprise  Overview  Cont 

176 

24.8 

12 

5  C2  Enterprise 

127 

14 

12 

6  C2  Enterprise  Cont 

260 

6.1 

12 

7  ESC  tailored 

230 

15.9 

12 

8  Space  and  Misile  Systems 

233 

23.4 

12 

9  Space  and  Misile  Systems 
Cont 

74 

31.5 

12 

10  Air  Armament  Systems 

327 

13.5 

12 

1 1  Air  Armament  Systems 

Cont. 

133 

31.1 

12 

12  Air  Armament  Systems 

Cont. 

321 

9.1 

12 

13  Air  Systems 

222 

0 

12 

14  Air  Systems  Cont 

398 

23.5 

12 

15  Air  Systems  Cont 

409 

9.2 

12 

16  Air  Systems  Cont 

196 

10.2 

12 

17  Other  Products 

352 

32.1 

12 

18  Other  Products  Cont 

169 

11.8 

12 

19  Summary 

128 

58.7 

8.1 
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Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  7 

Page  1  Summary  Exercise 

82 

50.9 

10.2 

Totals 

13547 

21.47 

11.63 

Avg.  Words/Module 

1935 

Usability  Items 

Module  2.1  Events  leading  to 
OSS&E 

When  closing  the  mishap  briefing  window,  system  generates  a 
close  window  icon  in  main  window  that  closes  main  window 
when  pressed. 

Module  2.2  Need  for  OSS&E 

AFPD  63-12  Link  Broken 

Module  2.2  Need  for  OSS&E 

API  63-1201  Link  Broken 

Module  2.2  Need  for  OSS&E 

AFMCI  63-1201  Link  Broken 

Module  2.3  Policy 
Documentation 

AF  Policy  Directive  63-12  link  broken 

Module  2.3  Policy 
Documentation 

AFMCI  Instruction  63-1201  link  broken 

Module  2.3  Policy 
Documentation 

Assurance  of  Operational  Safety...  link  broken 

Module  2.3  Policy 
Documentation 

http://escen. mitre. org:81/escn/ossetoolkit.html  link  broken 

Module  3.1  HHQ  Roles 

USAF/IL  link  broken 

Module  3.1  HHQ  Roles 

HQ  USAF/SE  link  broken 

Module  3.1  HHQ  Roles 

HQ  USAF/XO  link  broken 

Module  3.1  HHQ  Roles 

API  63-1201,  paragraph  2.1  link  broken 

Module  3.3  Chief  Engineer 

Government  Industry  Data  Exchange  Program  link  broken 

Module  4.2  Baselines  used  to 
manage... 

Interactive  exercise  window  loses  focus  after  generating 

Module  5.1  Disciplined 
Engineering  Process 

Interactive  exercise  window  loses  focus  after  generating 

Module  5.1  Disciplined 
Engineering  Process 

AFPD  90-9  link  broken 

Module  5.1  Disciplined 
Engineering  Process 

AFPD  90-901  link  broken 

Module  5.1  Disciplined 
Engineering  Process 

API  09-202  link  broken 

Module  5.1  Disciplined 
Engineering  Process 

AFMC  Pamphlet  63-104...  link  broken 

Module  5.1  Disciplined 
Engineering  Process 

API  99-101  link  broken 

Module  5.1  Disciplined 
Engineering  Process 

API  90-102  link  broken 

Module  5.6  Useful  Links 

AFPD  62-4  link  broken 

Module  5.6  Useful  Links 

AFPD  62-5  link  broken 

Module  5.6  Useful  Links 

API  91-103...  link  broken 
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Module  6.1  Documenting 
Compliance 

API  16-1002...  link  broken 

Module  6.1  Air  Armament 
Systems 

AF  Instruction  21-101 ...  link  broken 

Module  5.6  Other 
Products/Items 

Interactive  exercise  window  loses  focus  after  generating 

Module  6.8  Other  Useful 

Links 

API  21-201  link  broken 

Module  6.8  Other  Useful 

Links 

AFI  63-101  link  broken 

Module  6.8  Other  Useful 

Links 

AFI  63-104  link  broken 

Module  6.8  Other  Useful 

Links 

AFI  63-107  link  broken 

Module  6.8  Other  Useful 

Links 

AFI  91-103  link  broken 

Module  6.8  Other  Useful 

Links 

AFI  91-205  link  broken 

Module  6.8  Other  Useful 

Links 

AFI  99-105  link  broken 

Module  7.1  Summary 

Exercises 

Interactive  exercise  window  loses  focus  after  generating 

Total  Links 

67 

Broken  Links 

30 

Broken  Links  % 

45% 
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Course  Name: 

Advance  Concept  Technology  Demonstration 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  1 

Page  1 

477 

46.6 

10.9 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  2 

Page  1 

285 

0.7 

12 

Page  2 

368 

8.5 

12 

Page  3 

154 

21.6 

12 

Page  4 

431 

14.8 

12 

Page  5 

206 

19.5 

12 

Page  6 

398 

11.3 

12 

Page  7 

346 

22.8 

12 

Page  8 

670 

22.8 

12 

Page  9 

340 

15.9 

12 

Page  10 

405 

25.6 

12 

Page  1 1 

345 

23 

12 

Page  12 

198 

21.1 

12 

222 

45.2 

11 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  3 

Page  1 

177 

36.3 

11.6 

Page  2 

142 

12.6 

12 

Page  3 

156 

27 

12 

Page  4 

168 

25.9 

12 

Page  5 

326 

36.3 

12 

Page  6 

120 

35.6 

12 

Page  7 

306 

26.6 

12 

Page  8 

152 

56.2 

8.4 

Page  9 

157 

37.2 

11.4 

Page  10 

117 

42.1 

11.1 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  4 

Page  1 

276 

23 

12 

Page  2 

293 

11.2 

12 

Page  3 

196 

25.5 

12 

Page  4 

323 

20.1 

12 

Page  5 

205 

14.9 

12 

Page  6 

373 

18.9 

12 

92 


Page  7 

62 

30.8 

12 

Page  8 

50 

24.9 

12 

Page  9 

64 

16.4 

12 

Page  10 

190 

36.1 

12 

Page  1 1 

422 

18.9 

12 

Page  12 

217 

15.2 

12 

Page  13 

74 

37.8 

11.6 

Page  14 

122 

37.1 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  5 

Page  1 

226 

44.6 

11.3 

Page  2 

229 

38.9 

11.1 

Page  3 

230 

35 

12 

Page  4 

297 

42.3 

12 

Page  5 

238 

32.3 

12 

Page  6 

97 

44.9 

10.7 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  6 

Page  1 

280 

17.8 

12 

Page  2 

284 

18.6 

12 

Page  3 

143 

18.6 

12 

Page  4 

251 

26.2 

12 

Page  5 

196 

23.8 

12 

Page  6 

126 

5.3 

12 

Page  7 

402 

36.8 

12 

Page  8 

140 

41.8 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  7 

Page  1 

320 

28.6 

12 

Page  2 

97 

25.9 

12 

Page  3 

339 

30.1 

12 

Page  4 

258 

22.5 

12 

Page  5 

199 

10.6 

12 

Page  6 

557 

15.6 

12 

Page  7 

376 

21.1 

12 

Page  8 

118 

0 

12 

Page  9 

207 

39.8 

11 

Page  10 

128 

40.4 

11.3 

Page  1 1 

179 

22.6 

12 

Page  12 

289 

32 

12 

Page  13 

379 

41.8 

10.9 

93 


Page  14 

151 

32.4 

12 

Page  15 

134 

67.5 

7.5 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  8 

Page  1 

352 

15.6 

12 

Page  2 

276 

7.5 

12 

Page  3 

265 

11.3 

12 

Page  4 

297 

25.9 

12 

Page  5 

294 

33.6 

12 

Page  6 

369 

18.1 

12 

Page  7 

231 

30.5 

12 

Page  8 

344 

36.6 

12 

Page  9 

212 

32.2 

12 

Page  10 

280 

25 

12 

Page  1 1 

210 

23.1 

12 

Page  12 

98 

2.3 

12 

Page  13 

104 

60.1 

9 

Totals 

19735 

26.82 

11.73 

Avg.  Words/Module 

2467 

Module  2.1 

Joint  Requirements  Oversight  Council  (JROC)  link  broken 

Site  Failure 

AFIT  web  server  went  down  at  1550  hrs.  Down  all  weekend. 

2.6  Roles  and 
Responsibilities,  cont. 

JROC  Backgroung  Information  link  broken 

2.6  Roles  and 
Responsibilities,  cont. 

HQ  USAF/XOCW  link  broken  (Restricted) 

2.6  Roles  and 
Responsibilities,  cont. 

Air  Force  Materiel  Command  (AFMC)  link  broken 

3.1  The  Modernization 
Planning  Process,  cont. 

AFI  61-105,  Planning  for  S&T  link  broken 

3.2  Origins  of  ACTDs 

Joint  Service  sponsorship  link  does  not  return  to  calling  page 

3.2  Budget  subprocess 

AFMPP  link  does  not  return  to  calling  page 

6.3  AF  Guidance 

Issues,  cont. 

Air  Force  and  DOD  guidance  on  Ots  link  broken 

7.4  OSS&E 

AFMC  Responsibilities  (AFI  63-1201)  linnk  broken 

7.5  T&E  of  ACTDs 

AFPD  99-101,  para  5.6  link  broken 

7.5  T&E  of  ACTDs 

Operational  Utility  Evaluation  (OUE)  AFI  99-102,  para  2.3.3  link 
broken 

7.5  T&E  of  ACTDs 

AFI  99-101  para  3. 7/3. 8  Development,  Test,  and  Evaluation  link 
broken 

8.1  Transition  Planning 
Guidelines 

Transition  Planning  Guidelines  link  broken 

When  returning  to  Student  Report  Card  from  Summary  screens,  a 
new  frameset  is  created  in  the  right  frame  of  the  original  frameset 
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Course  Name:  Reducing  Acquisition  Response  Time 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  1 

Page  1 

128 

48.2 

9.9 

Page  2 

41 

42.3 

10.7 

Page  3 

65 

32.5 

12 

Page  4 

190 

43.6 

11.2 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  2 

Page  1 

141 

40.2 

12 

Page  2 

131 

28.5 

12 

Page  3 

38 

54.6 

9.4 

Page  4 

65 

17 

12 

Page  5 

156 

40.8 

9.7 

Page  6 

120 

40.6 

12 

Page  7 

100 

34.2 

12 

Page  8 

157 

37.9 

12 

Page  9 

104 

17 

12 

Page  10 

199 

38.9 

12 

Page  1 1 

287 

23.4 

12 

Page  12 

151 

36.5 

12 

Page  13 

207 

34.1 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  3 

Page  1 

236 

46.2 

11.4 

Page  2 

245 

33.4 

12 

Page  3 

234 

29.3 

12 

Page  4 

198 

38.4 

11.4 

Page  5 

141 

23.3 

12 

Page  6 

143 

34.8 

12 

Page  7 

401 

29.2 

12 

Page  8 

174 

35.9 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  4 

Page  1 

157 

47.8 

10.5 

Page  2 

90 

36.7 

11.6 

Page  3 

183 

37.5 

12 

96 


Page  4 

53 

52.8 

10 

Page  5 

56 

53.5 

10 

Page  6 

59 

52.2 

10.4 

Page  7 

253 

34.5 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  4 

Page  1 

51 

21 

12 

Page  2 

98 

34.2 

12 

Page  3 

74 

13.8 

12 

Page  4 

82 

20.8 

12 

Page  5 

79 

31.5 

11.7 

Page  6 

78 

17 

12 

Page  7 

88 

6.1 

12 

Page  8 

104 

30.1 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  4 

Page  1 

183 

19.9 

12 

Page  2 

84 

61.8 

7.2 

Page  3 

110 

42.5 

9.9 

Page  4 

97 

49.6 

10.2 

Page  5 

69 

49.5 

9.5 

Totals 

6100 

35.41 

11.35 

Avg.  Words/Module 

1525 

3.1  Commercial  Business 
Tools 

Activity  Based  Costing  link  broken 

3.1  Commercial  Business 
Tools 

Balanced  Scorecard  link  broken 

3.1  Commercial  Business 
Tools 

Benchmarking  link  broken 

3.1  Commercial  Business 
Tools 

Core  Competencies  link  broken 

3.1  Commercial  Business 
Tools 

Customer  Satisfaction  link  broken 

3.1  Commercial  Business 
Tools 

Cycle  Time  Reduction  link  broken 

3.1  Commercial  Business 
Tools 

Knowledge  Management  link  broken 

3.1  Commercial  Business 
Tools 

Mission  and  Vision  Statements  link  broken 

3.1  Commercial  Business 
Tools 

Pay  for  Performance  link  broken 

3.1  Commercial  Business 
Tools 

Reengineering  link  broken 

3.1  Commercial  Business 
Tools 

Scenario  Planning  link  broken 
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3.1  Commercial  Business 
Tools 

Shareholder  Value  Analysis  link  broken 

3.1  Commercial  Business 
Tools 

Strategic  Alliances  link  broken 

3.1  Commercial  Business 
Tools 

Strategic  Planning  link  broken 

3.1  Commercial  Business 
Tools 

Total  Quality  Management  link  broken 

3.1  Commercial  Business 
Tools 

Value  Chain  Analysis  link  broken 

3.5  Recommended 
Reading/Resources 

Portfolio  Management  For  New  Products  link  broken 

5.1  Cycle  Time  Reduction 
Recommendations 

Policy  Memorandum  link  broken 

5.1  Cycle  Time  Reduction 
Recommendations 

pathfinder  projects  link  broken 

Total  Links 

93 

Broken  Links 

19 

Broken  Links  % 

20% 
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Course  Name: 

Incentive  for  Reducing  Acquisition  Response  Time 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  1 

Page  1 

266 

46.5 

11.8 

Page  2 

41 

42.3 

10.7 

Page  3 

48 

51.3 

10 

Page  4 

229 

29.6 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  2 

Page  1 

266 

35.5 

12 

Page  2 

120 

11.3 

12 

Page  3 

107 

12.8 

12 

Page  4 

204 

10.9 

12 

Page  5 

142 

8.2 

12 

Page  6 

81 

23.3 

12 

Page  7 

129 

19.3 

12 

Page  8 

171 

5 

12 

Page  9 

189 

16.4 

12 

77 

0 

12 

125 

25.1 

12 

Page  12 

253 

28.1 

12 

Page  13 

31 

21.5 

12 

Page  14 

98 

42.4 

11.8 

Page  15 

86 

30.3 

12 

Page  16 

227 

36.7 

12 

Page  17 

198 

15.2 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  3 

Page  1 

138 

39 

12 

Page  2 

238 

46.5 

10.3 

Page  3 

165 

47.2 

10 

Page  4 

148 

40.1 

12 

Page  5 

210 

40.8 

12 

Page  6 

235 

29 

12 

Page  7 

198 

28.5 

12 

Page  8 

246 

10.2 

12 

Page  9 

194 

6.3 

12 

Page  10 

184 

34.8 

12 

Page  1 1 

211 

37.8 

12 
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Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  4 

Page  1 

137 

30.5 

12 

Page  2 

124 

26.2 

12 

Page  3 

195 

20.9 

12 

Page  4 

110 

34.5 

12 

Page  5 

149 

4.6 

12 

Page  6 

73 

21 

12 

Page  7 

153 

15.5 

12 

Page  8 

187 

16.8 

12 

Page  9 

37 

5.2 

12 

Page  10 

173 

25.9 

12 

Page  1 1 

259 

9.1 

12 

Page  12 

98 

28.8 

12 

Page  13 

203 

35.3 

12 

Page  14 

96 

2.8 

12 

Page  15 

303 

37 

12 

Page  16 

274 

21 

12 

Page  17 

283 

32.1 

12 

Page  18 

221 

38.8 

12 

Page  19 

224 

34.3 

12 

Page  20 

188 

42.7 

10.8 

Page  21 

126 

42.6 

10.3 

Page  22 

245 

21.6 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  5 

Page  1 

141 

40.5 

12 

Page  2 

164 

38.3 

11.8 

Page  3 

152 

20.4 

12 

Page  4 

201 

32.5 

12 

Page  5 

157 

38.1 

12 

Page  6 

149 

37.4 

12 

Page  7 

129 

4.8 

12 

Page  8 

117 

42.8 

12 

Page  9 

155 

20.1 

12 

Page  10 

195 

36.7 

12 

Page  1 1 

145 

18.7 

12 

Page  12 

39 

31.4 

12 

Page  13 

187 

34.6 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  6 

100 


Page  1 

266 

39.9 

10.2 

Page  2 

117 

21.5 

12 

Page  3 

242 

35.8 

12 

Page  4 

85 

45.1 

10.4 

Page  5 

25 

72.3 

6.2 

Page  6 

247 

34.4 

12 

Page  7 

301 

2.9 

12 

Page  8 

92 

16.5 

12 

Page  9 

234 

23.2 

12 

Page  10 

146 

25.7 

12 

Page  1 1 

182 

18.1 

12 

Page  12 

223 

16.2 

12 

Page  13 

175 

26 

12 

Totals 

13379 

27.41 

11.75 

Avg.  Words/Module 

2230 

6.1  Schedule  Based 
Incentives 

Constructing  Successful  business  incentives  link  broken.  Student 
warned  site  may  be  unavailable. 

Total  links 

119 

Broken  Links 

1 

Broken  Links  % 

1% 
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Course  Name:  Intelligence  in  Force  Modernization 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  1 

Page  1 

214 

25.7 

12 

Page  2 

110 

50.4 

9.4 

Page  3 

115 

39.7 

12 

Page  4 

138 

50.7 

10.4 

Page  5 

87 

56.9 

8.5 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  2 

Page  1 

382 

18.6 

12 

Page  2 

346 

24.2 

12 

Page  3 

81 

40 

10.7 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  3 

Page  1 

136 

1.8 

12 

Page  2 

43 

19 

12 

Page  3 

145 

18.7 

12 

Page  4 

240 

17.6 

12 

Page  5 

156 

20.5 

12 

Page  6 

125 

31.7 

12 

Page  7 

80 

58 

8.2 

Page  8 

242 

48 

9 

Page  9 

85 

37 

10.7 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  4 

Page  1 

148 

0 

12 

Page  2 

422 

20.6 

12 

Page  3 

179 

0 

12 

Page  4 

41 

61 

7.4 

Page  5 

352 

14.2 

12 

Page  6 

231 

5.5 

12 

Page  7 

196 

12.9 

12 

Page  8 

439 

21.1 

12 

Page  9 

365 

9.1 

12 

Page  10 

353 

6.8 

12 

Page  1 1 

411 

6.1 

12 

Page  12 

305 

10.1 

12 

102 


Page  13 

322 

5.6 

12 

Page  14 

126 

33.2 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  5 

Page  1 

161 

0 

12 

Page  2 

499 

32.1 

12 

Page  3 

291 

44.3 

10.8 

Page  4 

380 

43 

11 

Page  5 

140 

47.5 

10 

Page  6 

354 

45.6 

10.2 

Page  7 

42.5 

Page  8 

371 

57.3 

8.7 

Page  9 

154 

54.8 

9.7 

Page  10 

68 

0 

0 

Page  1 1 

315 

56.3 

9.7 

Page  12 

204 

46.3 

10.2 

Page  13 

96 

21.4 

12 

Page  14 

122 

21.3 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  6 

Page  1 

117 

27.5 

12 

Page  2 

358 

38.7 

12 

Page  3 

166 

30.6 

12 

Page  4 

487 

40.4 

11.5 

Page  5 

433 

34.6 

12 

Page  6 

60 

10.5 

12 

Page  7 

251 

28.4 

12 

Page  8 

100 

36.8 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  7 

Page  1 

84 

8.3 

12 

Page  2 

318 

12 

12 

Page  3 

423 

23.3 

12 

Page  4 

181 

39.6 

11.5 

Page  5 

437 

47 

11.4 

Page  6 

274 

27.7 

11.5 

Page  7 

676 

39.2 

11.9 

Page  8 

224 

43.7 

12 

Page  9 

249 

30.3 

12 

Page  10 

396 

0 

12 

Page  1 1 

207 

48.4 

9.2 

103 


434 

25.6 

12 

Page  13 

439 

23.7 

12 

Page  14 

182 

29.4 

12 

Page  15 

466 

20.8 

12 

Page  16 

98 

16.5 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  8 

Page  1 

45 

18.9 

12 

Page  2 

409 

21.4 

12 

Page  3 

545 

38.9 

12 

Page  4 

99 

0 

0 

Page  5 

491 

26.9 

12 

Page  6 

461 

14.5 

12 

Page  7 

144 

30.1 

12 

Page  8 

93 

23.7 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  9 

Page  1 

366 

30.5 

12 

Page  2 

220 

13.1 

12 

Page  3 

158 

15.5 

12 

Page  4 

52 

18.9 

12 

Page  5 

210 

30.4 

12 

Page  6 

241 

19.4 

12 

Page  7 

284 

20.5 

12 

Page  8 

510 

15.6 

12 

Page  9 

125 

39.5 

11.5 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  10 

Page  1 

467 

3.5 

12 

Page  2 

298 

0 

12 

Page  3 

294 

3.5 

12 

Page  4 

347 

27.2 

12 

Page  5 

272 

25.1 

12 

Page  6 

302 

12.7 

12 

Page  7 

533 

2.1 

12 

Page  8 

115 

35.7 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  11 

Page  1 

164 

19.1 

12 

104 


Page  2 

184 

2.3 

12 

Page  3 

212 

24.5 

12 

Page  4 

337 

17.9 

12 

Page  5 

367 

14 

12 

Page  6 

66 

20.9 

12 

Page  7 

146 

44.2 

11.4 

Page  8 

183 

29.2 

12 

Page  9 

239 

35.3 

12 

Page  10 

279 

40.9 

11.5 

Page  1 1 

187 

25.6 

12 

Page  12 

189 

0 

12 

Page  13 

126 

31.6 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  12 

Page  1 

92 

0.9 

12 

Page  2 

404 

10 

12 

Page  3 

354 

22.3 

12 

Page  4 

221 

0 

12 

Page  5 

244 

4.3 

12 

223 

0 

12 

Page  7 

359 

20.8 

12 

Page  8 

CO 

12 

Page  9 

102 

24.4 

12 

Page  10 

511 

16.3 

12 

Page  1 1 

168 

14.2 

12 

Page  12 

72 

25.4 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  13 

Page  1 

100 

10.7 

12 

Page  2 

227 

12.1 

12 

Page  3 

223 

23.8 

12 

Page  4 

236 

28.5 

12 

Page  5 

71 

29.8 

11.7 

Page  6 

178 

29.6 

12 

Page  7 

176 

27.2 

12 

Page  8 

141 

20.4 

12 

Page  9 

69 

30.3 

12 

Page  10 

384 

24.3 

12 

Page  1 1 

149 

62.9 

6.9 

Page  12 

140 

47.1 

10.9 

Page  13 

149 

23.7 

12 

Page  14 

254 

23.5 

12 

Total 

32648 

24.70 

11.43 
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Avg.  Words/Module 

2511 

Usability  Items 

1.3  AFIFM  COP 

Comminity  of  Practice  link  broken 

1.4  Summary 

Air  Force  is  spelled  Air  Free  in  1 .5 

2.2  New  Strategy 

https://www.afmc-mil.wpafb.af.mil/HQ- 
AFMS/IN/force  mod/0001 04strategy.htm  link  broken 

2.2  New  Strategy 

AFI  14-111  link  broken 

2.2  New  Strategy 

WSIIO  link  broken 

3.2  Requirements 

Generation  Process 

Chairman  JCS  Inst(CJCSI)  3170.1  link  broken 

3.2  Requirements 

Generation  Process 

AF  Policy  Directive  90-1 1  link  broken 

3.2  Requirements 

Generation  Process 

MPP  Phases,  Definition  Deskbook  link  broken 

3.2  Requirements 

Generation  Process 

MPP  Phases,  Documentation  Deskbook  link  broken 

3.2  Requirements 

Generation  Process 

MPP  Phases,  Validation  Deskbook  link  broken 

3.2  Requirements 

Generation  Process 

MPP  Phases,  Approval  Deskbook  link  broken 

3.2  Requirements 

Generation  Process 

AF  Instruction  10-601  Guidance  and...  link  broken 

3.2  Requirements 

Generation  Process 

AF  Instruction  10-601  Operational  Requirements...  link  broken 

3.3  The  Acquisition 
Management  System 

click  here  for  this  diagram  in  the  deskbook  library  link  broken 

3.3  The  Acquisition 
Management  System,  cont 

Interim  DoD  5000. 2-R  link  broken 

4.3  Other  Organization 
Supporting  IFM 

XOI  website  link  broken 

4.3  Other  Organization 
Supporting  IFM 

AFCA  webpagr  link  broken 

4.3  Other  Organization 
Supporting  IFM 

AIA  webpage  link  broken 

4.3  Other  Organization 
Supporting  IFM 

NAIC  website  link  broken 

4.4  US  Intelligence 
Community,  cont 

Director  of  Central  Intelligence  link  broken 

4.4  US  Intelligence 
Community,  cont 

Links  to  Intel  Community  link  broken 

4.4  US  Intelligence 
Community,  cont 

The  national  intelligence  community  organizations  affecting  IFM 
3rd  para  is  unintelligible 

5.1  Directives  and 

Guidance 

DoD  5000. 2-R  link  broken 

5.1  Directives  and 

Guidance 

CJCSI  3170.01  Requirements  Generation  link  broken 

5.1  Directives  and 

Guidance 

DIAR  55-3  link  broken 

5.1  Directives  and 

Guidance 

Defense  Planning  Guidance,  FY01-05  link  broken 
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5.1  Directives  and 

Guidance 

AFPD  10-6,  Mission  Needs  &  Operational  Requirements  link 
broken 

5.1  Directives  and 

Guidance 

AFI  10-601  Mission  Needs  &  Operational  Requirements 

Guidance  and  Procedures  link  broken 

5.1  Directives  and 

Guidance 

AFI  14-1 1 1 ,  Intelligence  support  to  the  AF  Acquisition  process 
link  broken  link  broken 

5.1  Directives  and 

Guidance 

DIAR  55-3  programmer  notes  visible  on  screen 

5.1  Directives  and 

Guidance 

Defense  Planning  Guidance,  FY01-05  programmer  notes  visible 
on  screen 

5.2  Threat  Information 
Integration  in  IFM 

Support  link  in  macromedia  arrow  requires  mouse  to  be  over  a 
letter  before  link  activates 

5.5  Roles  and 

Responsibilities  of  AF  Intel 

Air  Force  Instruction  14-111  link  broken 

6.1  Modeling  and 

Simulation  Overview,  cent. 

First  paragraph  of  Cost  and  Time  section  needs  grammar  check 
for  evaluating  vs  evaluation 

6.1  Modeling  and 

Simulation  Policy  and 
Guidance 

DoDD  5000.59  link  broken 

6.1  Modeling  and 

Simulation  Policy  and 
Guidance 

AFPD  16-10  link  broken 

6.1  Modeling  and 

Simulation  Policy  and 
Guidance 

AFI  16-1002  link  broken 

7.3  ISWG/TEM  cont. 

Intelligence  Support  planner...  link  broken 

7.5  Requirements  and 
Deficiencies,  cont. 

Intelligence  Support  planner...  link  broken 

9.2  Directives  and 

Guidance 

AFI  14-111  link  broken 

9.2  Directives  and 

Guidance 

https://www.afca.scottaf.mil/c4isp/  link  broken 

9.2  Directives  and 

Guidance 

Command,  Control,  Communications...  link  broken 

9.2  Directives  and 

Guidance 

Mandatory  procedures  for...  link  broken 

9.2  Directives  and 

Guidance 

Interoperability  and  Supportability...  link  broken 

9.4  Validation  and  Approval 
process 

Air  Force  C4ISP  prep  guide  link  broken 

9.5  Funding  and 
Implementation,  cont. 

Air  Force  C4ISP  prep  guide  link  broken 

10.2  Requirements  Process 
and  Procedures 

Community  online...  link  broken 

10.3  Characteristics  of  a 
well  written  requirement 

AFI  14-201  link  broken 

10.5  Directives  and 

Guidance  for  DoDIPP 

Air  Force  Policy  Directive  (AFPD)  14-1  link  broken 

10.5  Directives  and 

Guidance  for  DoDIPP 

Air  Force  Instruction  (AFI)  14-201  link  broken 

10.5  Directives  and 

Guidance  for  DoDIPP 

Title  X,  use  link  broken 
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11.1  GI&S  Definitions 

Text  Products  link  broken 

1 1 .3  GI&S  Requirements 
Process,  cent. 

DoD  5000.56  link  broken 

1 1 .3  GI&S  Requirements 
Process,  cont. 

DoD  5000. 2-R  link  broken 

1 1 .3  GI&S  Requirements 
Process,  cont. 

CJCSI3170.01A  link  broken 

1 1 .3  GI&S  Requirements 
Process,  cont. 

CJCSI  3901.01  link  broken 

1 1 .3  GI&S  Requirements 
Process,  cont. 

API  10-601  link  broken 

13.3  Key 

Players/Structures,  cont. 

AF/XPP  homepage  link  broken 

13.4  Summary 

http://www.xp.hq.af.mil/xpp/xpp.htm  link  broken 

13.4  Summary 

http://www.xp.hq.af.mil/xpx  link  broken 

Total  Links 

274 

Broken  Links 

55 

Broken  Links  % 

20% 
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Course  Name: 

AFRL  R&D  Case  File  Management  Course 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade 
Level 

Module  1 

Page  1 

570 

59.1 

8.9 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade 
Level 

Module  2 

Page  1 

497 

60 

8.4 

Page  2 

244 

64.7 

8.3 

Page  3 

325 

52.6 

11.3 

Page  4 

306 

57.7 

9 

Page  5 

237 

60.7 

7.7 

Page  6 

209 

66.5 

8.3 

Page  7 

129 

48.4 

11.5 

Page  8 

178 

35 

12 

Page  9 

99 

64.5 

9.1 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade 
Level 

Module  3 

Page  1 

51 

64.5 

7.4 

Page  2 

613 

49.6 

10.5 

Page  3 

181 

48.3 

10.4 

Page  4 

65 

65.7 

7.5 

Page  5 

205 

55.4 

10.2 

Page  6 

284 

34.4 

12 

Page  7 

119 

47.3 

10.8 

Page  8 

50 

57.9 

9.3 

Page  9 

131 

52.2 

11.3 

Page  10 

329 

58.9 

10.6 

Page  1 1 

122 

53.2 

10.7 

Page  12 

211 

54.1 

9.4 

Page  13 

174 

57.6 

8.5 

Page  14 

147 

63 

6.2 

Page  15 

293 

46.4 

9.8 

Page  16 

58 

44.7 

10.6 

Page  17 

77 

67.3 

7.7 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade 
Level 

Module  4 

Page  1 

91 

60 

9.4 

Page  2 

281 

49 

11 

109 


Page  3 

64 

53.6 

8.7 

Page  4 

212 

57.9 

8.6 

Page  5 

351 

70 

7.1 

Page  6 

241 

68.7 

9.1 

Totals 

7144 

56.03 

9.43 

Avg.  Words/Module 

1786 

Usability  Items 

2.1  Public  Law  and 
Official  Guidance 

AFRLI  61-201  link  broken 

2.5  Arm/Farm 
Responsibilities 

(AFRL  Form  2529a, b,c)  link  broken 

2.5  Arm/Farm 
Responsibilities 

(AFRL  Form  2529a, b,c)  page  1  link  broken 

2.5  Arm/Farm 
Responsibilities 

(AFRL  Form  2529a, b,c)  page  2  link  broken 

2.7  C  RAD  As,  etc 

Research  spelled  incorrectly  in  second  to  last  sentence 

3.1  Starting  a  Case 
File 

AFMC  Form  14  link  broken 

3.6  Close-out  and 
Retirement 

R&D  Case  File  Minimum  Requirements  link  broken 

3.8  Summary 

AFRLs  R&D  case  file  info  are  link  broken 

4.2  Checklist  Items 

AF  Form  2519  All  purpose  checklist,  AFRL  R&D  case  file 
overprint,  Pgl  link  broken 

4.2  Checklist  Items 

AF  Form  2519  All  purpose  checklist,  AFRL  R&D  case  file 
overprint,  Pg2  link  broken 

Total  Links 

53 

Broken  Links 

9 

Broken  Links  % 

17% 

no 


Course  Name: 

Commercial  Business  Approach 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  1 

Page  1 

546 

37.1 

12 

Page  2 

122 

37.7 

12 

Page  3 

132 

42.4 

10.6 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  2 

Page  1 

78 

38.4 

12 

Page  2 

206 

27.6 

12 

Page  3 

202 

42.1 

10.7 

Page  4 

112 

55.5 

8.5 

Page  5 

101 

56.7 

7.5 

Page  6 

117 

18.3 

12 

Page  7 

59 

26.2 

12 

Page  8 

163 

40.4 

11.2 

Page  9 

119 

43.1 

11 

Page  10 

127 

28.2 

12 

Page  1 1 

79 

38.7 

10.7 

Page  12 

121 

51.7 

8.9 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  3 

Page  1 

76 

43.7 

12 

Page  2 

15 

18.7 

12 

Page  3 

125 

6.1 

12 

Page  4 

67 

0 

0 

Page  5 

110 

51.1 

9.9 

Page  6 

68 

0 

0 

Page  7 

32 

63.6 

8.3 

Page  8 

105 

70.3 

7.3 

Page  9 

66 

2.3 

12 

Page  10 

153 

61 

9.4 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  4 

Page  1 

43 

8.8 

12 

Page  2 

180 

50.8 

11.7 

Page  3 

123 

0 

12 

Page  4 

204 

34.5 

11.9 

Ill 


Readability  Check: 

Words 

Flesch  Reading  Ease 

Grade  Level 

Module  5 

Page  1 

46 

38.3 

12 

Page  2 

83 

48 

8.8 

Page  3 

156 

42.8 

9.9 

Page  4 

106 

38.8 

11.4 

Page  5 

124 

62.9 

6.3 

Page  6 

93 

54.9 

9.6 

Totals 

4513 

36.36 

10.11 

Avg.  Words/Module 

903 

3.4  techniques _ Vivamus  link  broken 


Total  Links  31 

Broken  Links  _ ^ 

3 

Broken  Links  %  % 


Course  Name: 

Reformed  Supply  Su| 

Dport  Program 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  1 

Page  1 

107 

44 

11.5 

Page  2 

47 

46.8 

10.4 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  2 

Page  1 

320 

32.8 

12 

Page  2 

117 

13.7 

12 

Page  3 

107 

57.5 

9.2 

Page  4 

80 

58.5 

8.8 

Page  5 

101 

37.5 

11.9 

Page  6 

154 

37.5 

12 

Page  7 

91 

60.1 

9 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  3 

Page  1 

140 

23 

12 

Page  2 

54 

88.9 

3.8 

Page  3 

100 

29.3 

12 

Page  4 

159 

38.1 

12 

Page  5 

327 

39.4 

12 

Page  6 

81 

60.1 

9 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  4 

Page  1 

202 

5.8 

12 

Page  2 

179 

23.4 

12 

Page  3 

259 

29.8 

12 

Page  4 

176 

26.6 

12 

Page  5 

93 

60.1 

9 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  5 

Page  1 

58 

10.7 

12 

Page  2 

45 

32.8 

12 

Page  3 

144 

10.7 

12 

Page  4 

133 

47.5 

11.4 

Page  5 

187 

21.5 

12 

113 


Page  6 

104 

41.1 

12 

Page  7 

192 

32.4 

12 

Page  8 

241 

45.6 

11.4 

Page  9 

24 

37.9 

12 

Page  10 

143 

42.5 

12 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  6 

Page  1 

85 

18.9 

12 

Page  2 

93 

34.5 

12 

Page  3 

102 

33 

12 

Page  4 

63 

33.6 

11.7 

Page  5 

140 

22.3 

12 

Page  6 

124 

36.5 

11.8 

Page  7 

86 

50.4 

9.4 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  7 

Page  1 

108 

46.7 

12 

219 

19.1 

12 

279 

15.1 

12 

229 

23.9 

12 

394 

25 

12 

Page  6 

147 

60.1 

9 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  8 

Page  1 

109 

14.7 

12 

Page  2 

116 

5 

12 

Page  3 

198 

10 

12 

Page  4 

115 

35.6 

12 

Page  5 

78 

58.3 

8.4 

Readability  Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  9 

Page  1 

146 

22.1 

12 

Page  2 

147 

21.6 

12 

Page  3 

162 

38 

12 

Page  4 

220 

23.9 

12 

Page  5 

100 

23 

12 

Page  6 

102 

26.3 

12 

Page  7 

85 

60.1 

9 

Totals 

7812 

34.42 

11.29 

114 


Avg.  Words/Module 

868 

Usability  Items 

2.1  Required  Reading 

1.1  Scope  - 1.2  Background  link  broken 

2.2  Readings 

Deskbook  link  redirects 

2.2  Required  Reading 

1 .4  Overview  of  the  five  tenets  link  broken 

2.3  Required  Reading 

Data  Exchange  link  redirects 

2.3  Optional  Reading 

Archived  DoD  5000.2R  link  redirects 

3.1  Required  Reading 

Chapter  2:2. 1.1...  link  broken 

3.2  Required  Reading 

Chapter  2:  2.1.4...  link  broken 

3.3  Required  Reading 

Chapter  2:  2.1.6...  link  broken 

3.3  Optional  Reading 

RSSP  Guide...  link  broken 

4.1  Required  Reading 

Chapter  2:  2.2. . .  link  broken 

4.1  Required  Reading 

Attachment  9..  link  broken 

4.1  Optional  Reading 

Attachment  8...  link  broken 

4.2  Required  Reading 

Chapter  2:  2.2.3. . .  link  broken 

4.3  Required  Reading 

AFMCI  23-101  link  redirects 

4.3  Required  Reading 

Chapter  2:  2. 2. 4.1 ...  link  broken 

4.4  Required  Reading 

Chapter  2:  2.3. . .  link  broken 

4.4  Optional  Reading 

Attachment  20...  link  broken 

5.1  Required  Reading 

Chapter  2:  2.4. 1 . . .  link  broken 

5.1  Required  Reading 

Chapter  1:  1.3.2...  link  broken 

5.2  Required  Reading 

RSSP  Guide:  Attachment  14...  link  broken 

5.2  Required  Reading 

RSSP  Guide:  Sectuion  2.4.1 ...  link  broken 

5.2  Optional  Reading 

RSSP  Guide:  Attachment  7...  link  broken 

5.2  Required  Reading 

RSSP  Guide:  Attachment  32...  link  broken 

5.2  Required  Reading 

Chapter  2:  2. 4. 1.3. 5...  link  broken 

5.3  Required  Reading 

Chapter  2:  2. 2. 4. 1.1...  link  broken 

5.3  Required  Reading 

Chapter  2:  2. 4. 1.3. 6...  link  broken 

5.4  Required  Reading 

Chapter  2:  2.4.2. . .  link  broken 

6.1  Required  Reading 

Chapter  3:3.1. ..link  broken 

6.1  Required  Reading 

Chapter  3:  3.2...  link  broken 

6.1  Required  Reading 

Attachment  29...  link  broken 

6.2  Required  Reading 

Chapter  3:  3.3...  link  broken 

6.2  Optional  Reading 

RSSP  Guide:  Attachment  30...  link  broken 

6.2  Optional  Reading 

Chapter  2:  2. 3. 6. 2.1 ...  link  broken 

6.3  Required  Reading 

Chapter  3:  3.4...  link  broken 

6.3  Required  Reading 

Chapter  6:  6.3.1 ...  link  broken 

6.4  Required  Reading 

Chapter  3:  3.5...  link  broken 

6.5  Required  Reading 

Chapter  3:  3.6...  link  broken 

7.1  Required  Reading 

Chapter  4:  Tenet  3. . .  link  broken 

7.2  Required  Reading 

Chapter  4:  4.2. . .  link  broken 

7.2  Required  Reading 

Attachment  10...  link  broken 

7.2  Optional  Reading 

Attachment  33...  link  broken 

7.3  Required  Reading 

Chapter  4:  4.3. . .  link  broken 

8.1  Required  Reading 

Chapter  5:  Tenet  4...  link  broken 

115 


8.2  Required  Reading 

Chapter  5:  5.2...  link  broken 

8.2  Required  Reading 

Chapter  5:  5.3...  link  broken 

8.3  Required  Reading 

Chapter  5:  5.4...  link  broken 

8.3  Required  Reading 

RSSP  Guide:  Attachment  33.. 

.  link  broken 

8.4  Required  Reading 

Chapter  5:  5.6...  link  broken 

9.1  Required  Reading 

Chapter  6:  Tenet  5...  link  broken 

9.2  Required  Reading 

Chapter  6:  6.3...  link  broken 

9.2  Required  Reading 

Attachment  33...  link  broken 

9.3  Required  Reading 

Chapter  6:  6.4...  link  broken 

9.4  Required  Reading 

Chapter  6:  6.6...  link  broken 

Total  Links 

128 

Broken  Links 

53 

Broken  Links  % 

41% 

116 


Course  Name: 

Product  Support  Management  Planning 

Readability 

Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  1 

Page  1 

339 

54.5 

9.6 

Page  2 

279 

48.2 

10.3 

Page  3 

235 

22.6 

12 

Page  4 

242 

45.3 

10.2 

Page  5 

87 

20.9 

12 

Page  6 

94 

38.8 

10.7 

Readability 

Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  2 

Page  1 

181 

28.3 

12 

Page  2 

161 

29.1 

12 

Page  3 

169 

35.3 

12 

213 

40.4 

12 

Page  5 

217 

26.9 

12 

Page  6 

214 

37 

12 

Page  7 

208 

8.8 

12 

Page  8 

284 

40.7 

12 

Page  9 

121 

29.3 

12 

Page  10 

133 

7.6 

12 

Page  1 1 

152 

32.8 

12 

Readability 

Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  3 

Page  1 

408 

28.8 

12 

Page  2 

118 

31.9 

12 

Page  3 

42 

21.4 

12 

Page  4 

197 

21.6 

12 

Page  5 

381 

21.9 

12 

Page  6 

157 

40.1 

11.2 

Page  7 

258 

21.7 

12 

Page  8 

216 

23.9 

12 

Page  9 

225 

30.5 

12 

Page  10 

278 

18.5 

12 

Page  1 1 

244 

33 

12 

Page  12 

119 

28.1 

12 

Page  13 

46 

51.7 

8.9 

Page  14 

124 

43.7 

11 

Readability 

Words 

Flesch  Reading 

Flesch-Kincaid  Grade  Level 

117 


Check: 

Ease 

Module  4 

Page  1 

93 

29.9 

12 

Page  2 

111 

37.2 

12 

Page  3 

199 

46.8 

11.1 

Page  4 

356 

36.9 

11.3 

Page  5 

180 

29.3 

12 

Page  6 

129 

28.7 

12 

Page  7 

151 

38.4 

12 

Page  8 

433 

36.3 

12 

Page  9 

153 

43 

12 

Readability 

Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  5 

Page  1 

113 

31.5 

12 

Page  2 

128 

29.9 

12 

Page  3 

153 

12 

12 

Page  4 

136 

25.3 

12 

Page  5 

197 

30.8 

12 

Page  6 

125 

21.3 

12 

Page  7 

223 

12.8 

12 

Page  8 

99 

14.8 

12 

Page  9 

226 

37.3 

11.7 

Page  10 

185 

20.5 

12 

Page  1 1 

134 

32.7 

12 

Page  12 

79 

33.6 

12 

Page  13 

66 

74.6 

5.3 

Page  14 

192 

25.3 

12 

Readability 

Check: 

Words 

Flesch  Reading 
Ease 

Flesch-Kincaid  Grade  Level 

Module  6 

Page  1 

97 

39.3 

12 

Page  2 

183 

23.1 

12 

Page  3 

67 

52.5 

11.1 

Page  4 

191 

20.9 

12 

Page  5 

75 

13.1 

12 

Page  6 

149 

27.9 

12 

Page  7 

152 

38.8 

12 

Page  8 

178 

41.8 

11.7 

Page  9 

379 

24.8 

12 

Page  10 

138 

25.4 

12 

Totals 

11612 

31.25 

11.66 

Avg. 

Words/Module 

1935 

118 


Usability  Items 

A  new  frameset  is  constructed  inside  the  right  frame  when  returning  to  the 
student  report  card  from  the  summary  pages. 

3.1  PSMP 
Components 

PSMP  CoP  link  broken  -  Required  to  complete  the  exercise 

6.3  Reclamation, 
cont 

process  out  description  has  redundant  wording  -  "customer" 

Total  Links 

50 

Broken  Links 

1 

Broken  Links% 

2% 

119 


Course  Name: 

Integrated  Product  Support  Course 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  1 

Page  1 

525 

51.5 

10.7 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  2 

Page  1 

328 

6.8 

12 

Page  2 

461 

8.6 

12 

Page  3 

119 

12.1 

12 

Page  4 

108 

35.7 

12 

Page  5 

162 

44.9 

9.4 

Page  6 

82 

19.3 

12 

Page  7 

125 

2.4 

12 

Page  8 

49 

14.8 

12 

Page  9 

311 

7.1 

12 

Page  10 

488 

11.8 

12 

Page  1 1 

362 

13.4 

12 

Page  12 

428 

29.7 

12 

207 

11.6 

12 

176 

3.2 

12 

307 

16.7 

12 

259 

16.5 

12 

82 

33.6 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  3 

Page  1 

178 

6.6 

12 

Page  2 

149 

10.6 

12 

Page  3 

151 

33.4 

12 

Page  4 

183 

26.5 

12 

Page  5 

267 

21.1 

12 

Page  6 

213 

10.8 

12 

Page  7 

280 

20.4 

12 

Page  8 

183 

23.5 

12 

Page  9 

296 

25.9 

12 

Page  10 

157 

33.1 

12 

Page  1 1 

338 

30.8 

12 

Page  12 

181 

40.8 

12 

120 


Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  4 

Page  1 

140 

0 

12 

Page  2 

133 

0 

12 

Page  3 

251 

15.6 

12 

Page  4 

135 

13 

12 

Page  5 

96 

8.8 

12 

Page  6 

107 

4 

12 

Page  7 

117 

47.7 

10.8 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  5 

Page  1 

193 

13 

12 

Page  2 

150 

37 

11.8 

Page  3 

258 

27.4 

12 

Page  4 

73 

61.3 

6.8 

Page  5 

102 

10.9 

12 

Page  6 

169 

11.6 

12 

Page  7 

300 

9.1 

12 

Page  8 

257 

14.4 

12 

Page  9 

183 

1.9 

12 

Page  10 

248 

19.5 

12 

Page  1 1 

209 

25.3 

12 

Page  12 

87 

30.1 

12 

Page  13 

97 

50.4 

9.4 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  6 

Page  1 

247 

8.2 

12 

Page  2 

223 

6.8 

12 

Page  3 

63 

0 

12 

Page  4 

154 

37.6 

11.9 

167 

21.9 

12 

Page  6 

241 

12 

Page  7 

301 

14.9 

12 

Page  8 

216 

8.5 

12 

Page  9 

275 

25.6 

12 

Page  10 

199 

15.7 

12 

Page  1 1 

249 

11.8 

12 

Page  12 

291 

10.3 

12 

Page  13 

244 

16 

12 

Page  14 

78 

48.6 

9.5 

121 


Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  7 

Page  1 

132 

16.5 

12 

Page  2 

325 

22.3 

12 

Page  3 

319 

5.4 

12 

Page  4 

182 

23.2 

12 

Page  5 

109 

23.5 

12 

Page  6 

112 

13.8 

12 

Page  7 

114 

27.1 

12 

Page  8 

301 

12.7 

12 

Page  9 

107 

41.6 

11.8 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  8 

Page  1 

15.9 

12 

Page  2 

15.5 

12 

Page  3 

165 

33.2 

12 

Page  4 

221 

54.5 

8.3 

Page  5 

312 

43.7 

11 

84 

21.4 

12 

211 

45.4 

9.7 

242 

43 

10.9 

270 

48 

10 

107 

28.2 

12 

138 

33.3 

11.9 

85 

48.6 

8.9 

246 

27.3 

12 

322 

25.8 

12 

Page  15 

449 

22.4 

12 

Page  16 

240 

23.1 

12 

Page  17 

202 

14.6 

12 

Page  18 

142 

37.4 

11.2 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  9 

Page  1 

190 

19.1 

12 

Page  2 

244 

8.6 

12 

Page  3 

311 

29.5 

12 

Page  4 

310 

17.4 

12 

Page  5 

111 

29.4 

12 

Page  6 

261 

24.8 

12 

Page  7 

256 

18.1 

12 

Page  8 

234 

22.2 

12 

122 


Page  9 

256 

13.5 

12 

Page  10 

219 

31.3 

12 

Page  1 1 

125 

36.7 

12 

Page  12 

467 

11.7 

12 

Page  13 

109 

41.5 

10.9 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  10 

Page  1 

297 

27 

12 

Page  2 

247 

19.1 

12 

Page  3 

291 

18.8 

12 

Page  4 

253 

21.9 

12 

Page  5 

116 

13.6 

12 

Page  6 

261 

26.8 

12 

Page  7 

137 

45.6 

8.3 

Page  8 

225 

0 

12 

Page  9 

213 

11 

12 

Page  10 

239 

13.9 

12 

Page  1 1 

185 

31.1 

12 

Page  12 

136 

60.8 

9 

Page  13 

403 

7.8 

12 

Page  14 

333 

24.3 

12 

Page  15 

225 

10 

12 

Page  16 

274 

35.6 

12 

Page  17 

370 

11.6 

12 

Page  18 

352 

11.6 

12 

Page  19 

126 

36.1 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  11 

Page  1 

236 

15.3 

12 

Page  2 

174 

18.7 

12 

Page  3 

299 

3.5 

12 

Page  4 

184 

13.8 

12 

Page  5 

270 

2 

12 

Page  6 

333 

8.6 

12 

Page  7 

196 

0 

12 

Page  8 

271 

17.9 

12 

Page  9 

251 

6.9 

12 

Page  10 

361 

18.8 

12 

Page  1 1 

216 

6.3 

12 

Page  12 

267 

0 

12 

Page  13 

294 

5.7 

12 

Page  14 

292 

1.6 

12 

123 


Page  15 

344 

8.4 

12 

Page  16 

181 

16.5 

12 

Page  17 

82 

39.3 

11.5 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  12 

Page  1 

206 

0 

12 

Page  2 

230 

15 

12 

Page  3 

261 

18.2 

12 

Page  4 

167 

0 

12 

Page  5 

282 

0.1 

12 

Page  6 

113 

10.4 

12 

Page  7 

145 

12.2 

12 

Page  8 

126 

9.9 

12 

Page  9 

66 

60.1 

9 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  13 

Page  1 

278 

13.4 

12 

Page  2 

491 

21.2 

12 

121 

32.4 

12 

Page  4 

116 

22.6 

12 

Page  5 

238 

15.3 

12 

Page  6 

336 

34.5 

12 

Page  7 

269 

33 

12 

Page  8 

321 

43.1 

10.9 

Page  9 

190 

0 

12 

Page  10 

430 

5.4 

12 

Page  1 1 

271 

20.2 

12 

Page  12 

106 

60.1 

9 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  14 

Page  1 

96 

0 

12 

Page  2 

251 

25.2 

12 

Page  3 

179 

22.1 

12 

Page  4 

284 

13.1 

12 

Page  5 

239 

25.4 

12 

Page  6 

308 

36.7 

12 

Page  7 

210 

14.8 

12 

Page  8 

274 

42.2 

12 

Page  9 

182 

11.9 

12 

Page  10 

296 

38.1 

12 

124 


Page  1 1 

319 

31.9 

12 

Page  12 

221 

27.4 

12 

Page  13 

310 

40.3 

12 

Page  14 

479 

10.5 

12 

Page  15 

308 

4.1 

12 

Page  16 

217 

9.1 

12 

Page  17 

73 

60.1 

9 

Totals 

39982 

21.30 

11.73 

Avg.  Words/Module 

2856 

Usability  Items 

2.1  Purpose  of 

Material 

Management 

DoD  4104...  link  broken 

2.2  Org  Chart 

Aeronautical  Systems  link  broken  on  description  page 

2.2  Org  Chart 

Air  Armament  Systems  link  broken 

2.2  Org  Chart 

Space  and  Missile  Systems  link  broken  on  description  page 

2.2  Org  Chart 

Oklahoma  City  Air  Logistics  Center  link  broken  on  description  page 

2.2  Org  Chart 

Sacramento  Air  Logistics  Center  link  broken  on  description  page 

2.2  Org  Chart 

San  Antonio  Air  Logistics  Center  link  broken  on  description  page 

2.2  Org  Chart 

Air  Force  Flight  Test  Center  link  broken  on  description  page 

2.2  Org  Chart 

Phillips  Laboratory  link  broken  on  AFRL  page 

2.2  Org  Chart 

AMARC  link  broken  on  description  page 

2.2  Org  Chart 

CASC  link  broken  on  cataloging  and  Standardization  page 

2.2  Org  Chart 

DLIS  link  broken  on  cataloging  and  Standardization  page 

2.2  Org  Chart 

AGMC  link  broken  on  cataloging  and  Standardization  page 

2.2  AFMC 
Organization,  cont 

Test  and  Evaluation  Business  Area  link  broken 

2.2  AFMC 
Organization,  cont 

Product  Support  Business  Area  link  broken 

2.2  AFMC 
Organization,  cont 

ISAG  Business  Area  link  broken 

2.2  AFMC 
Organization,  cont 

IM  Business  Area  link  broken 

2.2  AFMC 
Organization,  cont 

l&S  Business  Area  link  broken 

2.3  Principle  Tenets 
of  Materiel 
Management 

AFMC  mission  link  broken 

2.5  Key  positions... 

Deputy  Chief  of  Staff,  Installations  and  logistics  link  broken 

2.5  Key  positions... 

DoDD  5000.1  Defense  Acquisition  link  broken 

2.5  Key  positions... 

MMT-Material  Management  teams  link  broken 

3.4  Agile  Combat... 

Global  Engagement  link  restricted 
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3.4  Agile  Combat... 

Global  Engagement  Series  link  broken 

4.1  Introduction 

DoDD  5000.1  link  broken 

4.1  Introduction 

DoDD  5000.2-R  link  broken 

5.1  Introduction 

DoD  5000.2  link  redirects 

5.8  Other  Useful 

Links 

Air  Force  Supply  Publication  link  broken 

6.1  Introduction 

Department  of  Defense  Manuals  link  broken 

6.1  Introduction 

AFMC  Manual  23-3,  Cataloging  and  Standardization  link  broken 

6.1  Introduction 

Defense  Logistic  Information  Service  link  broken 

6.1  Introduction 

DoD  Cataloging  link  broken 

6.6  Cataloging 

System 

Federal  Logistic  Information  System  (FLIS)  link  broken 

6.7  Key  Participants 

Defense  Logistic  Information  System  (DLIS)  link  broken 

6.7  Key  Participants 

Cataloging  and  Standardization  Center  (CASC)  link  broken 

6.7  Key  Participants 

Directorate  of  Air  Force  (DLIS-F)  link  broken 

6.7  Key  Participants 

HQAFMC/LGI  link  broken 

6.7  Key  Participants 

HQ  AFMC/LGIA  link  broken 

6.7  Key  Participants 

HQ  AFMC/SC  link  broken 

7.2  Items 

Managed... 

D062  link  broken 

7.2  Items 

Managed... 

D041  link  broken 

7.2  Items 

Managed... 

D041  recoverable  consumption...  link  broken 

8.3  Commodity  Buy 
Process,  cont 

Materiel  Support  Division  (MSD)  link  broken 

8.4  Buy  Process 

Tools 

Lightning  bolt  initiative  #10  link  broken 

8.7  Other  Useful 

Links 

Air  Force  Policy  Directive  23-1...  link  broken 

8.7  Other  Useful 

Links 

Air  Force  Instruction  23-102...  link  broken 

8.7  Other  Useful 

Links 

Aie  Force  Instruction  23-103...  link  broken 

8.7  Other  Useful 

Links 

AFMC  Polict  Directive  23-1...  link  broken 

9.2  Purpose  of 

DREP  and  CREP 

Air  Force  Materiel  Command...  link  broken 
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9.3  Repair  Process 
Steps,  cont 

EXPRESS  link  broken 

9.7  Other  Useful 

Links 

Agile  (LEAN)  Logistics  homepage  link  broken 

10.2  IWSM 

Program  Master  List  link  broken 

10.4  Total  Ownership 
Costs 

Total  Ownership  Costs  (R-TOC)  link  broken 

10.5  Key  Tools  and 
Information  Systems 

Weapon  System  Management  Information  System  (WSMIS)  link 
broken 

10.5  Key  Tools  and 
Information  Systems, 
cont 

AFI  21-103  link  broken 

10.5  Key  Tools  and 
Information  Systems, 
cont 

SEMR  link  broken 

10.6  Key 

Interfaces... 

Maintenance  Requirements  Review  Board  (MRRB)  link  broken 

10.6  Key 

Interfaces... 

MAJCOM  days  link  broken 

12.1  Introduction 

Joint  Logistics  Commanders  (JLC's)  link  broken 

12.4  DLA  Support  to 
the  Services,  cont 

Defense  National  Stockpile  Center  link  broken 

12.4  DLA  Support  to 
the  Services,  cont 

Defense  Distribution  Systems  Center  (DDSC)  link  broken 

13.1  Introduction 

Agile  Combat  Support  link  broken 

13.2  Contractor 
Logistics  Support 
(CLS) 

Department  of  Defense  Logistics  Strategic  plan  link  broken 

13.2  Contractor 
Logistics  Support 
(CLS) 

Reliability  Based  Logistics  (RBL)  link  broken 

13.3  The  Foreign 
Military  Sales 

Program 

1998  DoD  Logistics  Strategic  Plan...  link  broken 

13.3  The  Foreign 
Military  Sales 

Program,  cont 

Air  Force  Security  Assistance  Center  link  broken 

13.4  Commercial 

Items  for  Military 
Support 

"Defense  Acquisition  (para  4.2.2)"  link  broken 

13.6  Other  Useful 

Links 

DoD  5000.1,  Defense  Acquisition  link  broken 

13.6  Other  Useful 

Links 

DoD  5000. 2-R  Mandatory  Procedures...  link  broken 

14.1  Introduction 

AFMCR  65-9  link  broken 
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14.3  Types  of 
Reclamation 

API  21-104  link  broken 

14.3  Types  of 
Reclamation,  cont 

API  23-204  link  broken 

14.3  Types  of 
Reclamation,  cont 

APMCR  65-9  link  broken 

14.3  Types  of 
Reclamation,  cont 

APMCR  65-9  link  broken 

14.3  Types  of 
Reclamation,  cont 

APMCR  65-9  link  broken 

14.3  Types  of 
Reclamation,  cont 

API  21-104  link  broken 

14.4  Disposal... 

Defense  Demilitarization  Manual  link  broken 

Exercise 

The  GO  button  does  not  appear  after  completing  the  exercise 

Total  Links 

324 

Broken  Links 

72 

Broken  Links% 

22 

128 


Course  Name: 

Introduction  to  Evolutionary  Acquisition 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  1 

Page  1 

321 

33 

12 

Page  2 

296 

42.3 

12 

Page  3 

544 

18.7 

12 

Page  4 

403 

0.8 

12 

Page  5 

172 

0 

12 

Page  6 

294 

52.2 

9.9 

Page  7 

429 

18.6 

12 

Page  8 

320 

29.8 

12 

Page  9 

951 

54.2 

10.5 

Page  10 

430 

28.7 

12 

Page  1 1 

119 

41 

11.6 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  2 

Page  1 

169 

15.2 

12 

Page  2 

584 

33.9 

12 

Page  3  Interactive  Exercise 

632 

28 

12 

Page  4 

453 

31.4 

12 

Page  5  Sys  Eng.  IPT 

585 

27.2 

12 

Page  6  Test  Eval  IPT 

174 

54.1 

9.1 

Page  7  Logistics  IPT 

177 

56.4 

8.1 

Page  8  Fin  Man  IPT 

108 

55.4 

7 

Page  9  War  Fighter  Rep 

255 

51.8 

10 

Page  10  New  Email 

123 

38.1 

10 

Page  1 1  Refine  Inc  1 

495 

32.2 

12 

Page  12  Sys  Eng  Assessment 

521 

20.4 

12 

Page  13  Test/Eval  Assessment 

158 

54.1 

9.1 

Page  14  Log  Assessment 

155 

56.4 

8.1 

Page  15  Fin  Man  Assessment 

98 

53.8 

7.4 

Page  16  Cont.  Assessment 

101 

55 

8.3 

Page  17  Warfighter 

Assessment 

249 

51.8 

10 

Page  18  Email 

122 

41.3 

9.5 

Page  19  Ref  Inc  2 

389 

22.9 

12 

Page  20  Sys  Eng  Assessment 

447 

19.3 

12 

Page  21Test/Eval  Assessment 

139 

54.1 

9.1 

Page  22  Log  Assessment 

130 

56.4 

8.1 

Page  23  Fin  Assessment 

88 

52 

7.8 

Page  24  Contract  Assessment 

95 

55 

8.3 

Page  25  New  Mail 

122 

40.5 

9.7 
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Page  26  Sys  Eng  Assessment 

319 

18 

12 

Page  27  Test/Eval  Assessment 

119 

54.1 

9.1 

Page  28  Log  Assessment 

104 

56.4 

8.1 

Page  29  Fin  Assessment 

78 

50 

8.4 

Page  30  Contract  Assessment 

89 

55 

8.3 

Page  31  New  Mail 

117 

44.7 

9 

Page  32 

361 

28.7 

12 

Page  33 

395 

38.7 

10.8 

Page  34 

358 

26.4 

12 

Totals 

12788 

38.84 

10.34 

Avg.  Words/Module 

6394 

Usability  Items 

1.2  Policy  on-line  References 

DoDD  5000.1  "Defense  Acquisition"  link  broken 

1.2  Policy  on-line  References 

DoDD  5000.1  word  version  link  broken 

1.2  Policy  on-line  References 

DoDD  5000. 2-R...  link  broken 

1.2  Policy  on-line  References 

DoDD  5000. 2-R...  word  version  link  broken 

1.2  Policy  on-line  References 

Office  of  Management  and  Budget  circular  A-109  link  broken 

Total  Links 

43 

Broken  Links 

5 

Broken  Links% 

12% 
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Course  Name: 

Modification  Management 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  1 

Page  1 

370 

22.3 

12 

Page  2 

26 

50.4 

9.4 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  2 

Page  1 

55 

50 

10.6 

Page  2 

106 

7.3 

12 

Page  3 

259 

16.5 

12 

Page  4 

38 

60.7 

7.7 

Page  5  Kinds-Perm 

75 

0 

12 

Page  6  Perm-Safe 

46 

0 

12 

Page  7  T1  Mods 

162 

15.5 

12 

Page  8 

97 

17.9 

12 

Page  9 

90 

24.4 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  3 

Page  1 

56 

28.7 

12 

Page  2 

89 

24.3 

12 

Page  3 

225 

18.9 

12 

Page  4 

140 

15 

12 

Page  5 

74 

24.9 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  4 

Page  1 

60 

0 

12 

Page  2 

94 

23.2 

12 

Page  3 

53 

30.6 

12 

Page  4 

44 

11.4 

12 

Page  5 

206 

30.6 

12 

Page  6 

186 

32.6 

12 

Page  7 

128 

24.1 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid 
Grade  Level 

Module  5 

Page  1 

40 

33.9 

12 

Page  2 

76 

5.8 

12 
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Page  3 

305 

23.5 

12 

Page  4 

111 

19.5 

12 

Page  5 

111 

22.6 

12 

Page  6 

181 

13 

12 

Page  7 

90 

28.5 

12 

Page  8 

136 

39.5 

10.2 

Readability  Check: 

Words 

Plesch  Reading  Ease 

Plesch-Kincaid 
Grade  Level 

Module  6 

Page  1 

109 

30.1 

12 

Page  2 

161 

35.5 

12 

Page  3 

299 

28 

12 

Page  4 

138 

26 

12 

Page  5 

303 

21.3 

12 

Page  6 

107 

11.9 

12 

Readability  Check: 

Words 

Plesch  Reading  Ease 

Plesch-Kincaid 
Grade  Level 

Module  7 

Page  1 

94 

57.3 

8.5 

Page  2 

139 

26.5 

12 

Page  3 

126 

16.3 

12 

Page  4 

43 

64.4 

7.8 

Page  5 

133 

18.1 

12 

Page  6 

37 

59.4 

8 

Page  7 

61 

41 

10 

Page  8 

125 

21.6 

10.8 

Readability  Check: 

Words 

Plesch  Reading  Ease 

Plesch-Kincaid 
Grade  Level 

Module  8 

Page  1 

51 

69.3 

7.6 

Page  2 

185 

11.3 

12 

Page  3 

41 

0 

12 

Page  4 

48 

26.1 

11.9 

Page  5 

46 

36.7 

11.9 

Page  6 

203 

24 

12 

Page  7 

45 

28 

12 

Page  8 

130 

18.8 

12 

Totals 

6353 

26.17 

11.44 

Avg.  Words/Module 

794 

2.1 

API  63-1101  link  broken 

T-1  Modifications 

temporarily  is  misspelled  temporily 

A  complete  new  frameset  is  generated  in  the  body  frame  when 
returning  to  student  report  card 

3.1 

API  63-1101  link  broken 
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3.3 

API  63-1101,  Attachments  link  broken 

3.3 

Special  Processing  link  broken 

Module  4 

API  63-1101  link  broken 

5.1 

APPD  63-1 1  ahead  link  broken 

5.1 

API  63-1101  link  broken 

5.1 

DoD  5000.2-R  next  exit  link  broken 

5.1 

API  65-601  Vol.  1  link  broken 

5.2 

APPD  63-11  link  broken 

5.2 

API  63-1101  link  broken 

Text  presented  when  rolling  over  AP  Perm  3525  says  AP  Perm  3225 
instead 

Module  6 

Learning  point  5  is  ambiguous 

7.2 

API  63-1101  modification  Management,  Section  3.9  link  broken 

7.4  Summary 

Outcome  5  says  modes  instead  of  mods 

8.2 

API  63-1101  link  broken 

8.2 

API  63-1101  link  broken 

Module  8  Quiz 

Question  8.1.1  is  an  answer  and  not  a  question 

Some  Pop-up  windows  do  not  have  close  buttons  on  the  screen 

Total  Links 

140 

Broken  Links 

14 

Broken  Links% 

10% 
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Course  Name: 

Earned  Value  Management  System 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  1 

Page  1 

231 

34.7 

12 

Page  2 

207 

26.5 

12 

Page  3 

190 

36.4 

12 

Page  4 

290 

26.5 

12 

Page  5 

93 

11.5 

12 

Page  6 

214 

34.7 

12 

Page  7 

214 

17.8 

12 

Page  8 

121 

15.5 

12 

Page  9 

288 

14.9 

12 

Page  10 

260 

29.2 

12 

Page  1 1 

167 

13.6 

12 

Page  12 

119 

31.3 

12 

Page  13 

74 

28.9 

12 

Page  14 

308 

15 

12 

Page  15 

329 

21.2 

12 

Page  16 

140 

16.5 

12 

Page  17 

158 

15.2 

12 

Page  18 

192 

41.2 

11.2 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  2 

Page  1 

170 

40.3 

12 

Page  2 

195 

56.4 

9.3 

Page  3 

139 

35.1 

12 

Page  4 

115 

25.6 

12 

Page  5 

139 

41.2 

11.5 

Page  6 

165 

54.3 

9.2 

Page  7 

84 

43.7 

10.5 

Page  8 

190 

62.4 

8.9 

Page  9 

157 

49.5 

10.6 

Page  10 

127 

34.9 

12 

Page  1 1 

69 

15.6 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  3 

Page  1 

101 

36.9 

11.3 

Page  2 

180 

36.9 

12 

Page  3 

153 

34.8 

12 

Page  4 

126 

29.1 

12 
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Page  5 

115 

53.4 

9.8 

Page  6 

55 

45.5 

10.1 

Page  7 

57 

55.9 

8.8 

Page  8 

204 

40.5 

11.6 

Page  9 

194 

30 

12 

Page  10 

194 

14.3 

12 

Page  1 1 

180 

19 

12 

Page  12 

127 

23.2 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  4 

Page  1 

231 

32.4 

12 

Page  2 

219 

40.2 

11.8 

Page  3 

150 

44.5 

11.7 

Page  4 

105 

34.2 

12 

Page  5 

146 

41.8 

11.3 

Page  6 

255 

49.6 

11.5 

Page  7 

211 

25.5 

12 

Page  8 

263 

48.8 

10 

Page  9 

255 

30.7 

12 

Page  10 

232 

31.3 

11.4 

Page  1 1 

177 

32.9 

12 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
Level 

Module  5 

Page  1 

167 

20.6 

12 

Page  2 

247 

23.9 

12 

Page  3 

286 

33.6 

12 

Page  4 

194 

29.2 

12 

Page  5 

71 

16 

12 

Page  6 

244 

29.8 

12 

Page  7 

158 

46 

10.9 

Page  8 

184 

31.3 

12 

Page  9 

294 

30.4 

12 

Page  10 

179 

37.9 

11.1 

Readability  Check: 

Words 

Flesch  Reading  Ease 

Flesch-Kincaid  Grade 
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Single  Manager  Educational...  link  broken 

2.4  POM  Schedule 

AFMC/XPP  POM  Home  Page  link  broken 

3.1  PSMA  Resource 

Allocation... 

FY05  AFOM  Drafr  Guidance  link  broken 

3.7  Other  Useful  References 
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AFMC  Cost  Division  home  page  link  broken 
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